Here's the recording from our “Office Hours” session on 2019-12-04.
We hold public “Office Hours” every Wednesday at 11:30am PST to answer questions on all things DevOps/Terraform/Kubernetes/CICD related.
These “lunch & learn” style sessions are totally free and really just an opportunity to talk shop, ask questions and get answers.
Register here: cloudposse.com/office-hours
Basically, these sessions are an opportunity to get a free weekly consultation with Cloud Posse where you can literally “ask me anything” (AMA). Since we're all engineers, this also helps us better understand the challenges our users have so we can better focus on solving the real problems you have and address the problems/gaps in our tools.
Machine Generated Transcript
Let's get started.
Welcome to both of course, our presents.
December 4th 2019 my name is Eric Osterman I am and I'll be leading the right my screens going nuts.
Now be leading conversation.
I'm CEO and founder of cloud posse where DevOps accelerator we help startups own their infrastructure in record time by building it for you.
And then showing you the ropes.
For those of you new to the call the format of this call is very informal.
My goal is to get your questions answered.
Feel free to unseat yourself at any time if you want to jump, jump in and participate.
We go to these calls every week will automatically post a video of this recording in the office hours channel as well as follow up email.
So you can share it with your team.
You want to share something in private.
And we can temporarily temporarily suspend the recording.
With that said, let's kick this off.
Here's the agenda for today.
First thing is Igor has joined.
He's part of the cloud posse team.
He's been with me for a very long time.
And he's going to share a little bit about the ThoughtWorks.
This is really cool if you haven't seen it before.
It's a great way to stay on top of what's happening in our industry.
And then if we have some time, we'll go or reinvent announcements some relevant ones that are obvious.
Somebody is a microphone there.
Vincent I guess.
Good to meet you Vincent.
There we go.
Sorry about that.
And then let's get started.
So first thing I just want to see if anybody has any pressing questions that you need answered.
This can be related to cloud posse stuff cloud boss terrible modules terraforming general Kubernetes or just general architecture questions related DevOps and cloud.
Yes they'll hear it.
And I trust you to have a good question.
How do you guys kind of do am on Ethiopia's how do we manage our subnet allocations will be another way.
Mute my notifications here.
I'm getting slack bond done.
We spent a fair bit of time dealing with this.
Now it would've been like a year or two.
But he came back up again earlier this year when we wanted to create a subnet architecture that spanned multiple eight US accounts and was to some degree future proof.
So that we could continue adding accounts.
It's a challenge because if you want to support peering between accounts, and BBC you need to think of that ahead of time.
So the best example, I have for that would be how we're actually doing it right now.
Woops let's get out of fullscreen mode here.
So under the cloud policy reference architectures.
We implement one strategy.
You've implemented the reference architecture.
So you undoubtedly interacted with it.
But you might not have just known it.
Let's see if I remember exactly where that cider so but Terraform provides a bunch of subnet calculation functions where interpretations.
And that's what we used to do.
To do this.
I'm taking I'm guessing because you're advanced at Terraform.
I'm guessing your question might be deeper than this.
What we basically did is we took a large.
I think slash 8.
And then we divided it evenly across the number of potential accounts that we would have.
And then gave each one of those accounts that CIDR block and then within that account, then we further subdivided for the species there in one of the things we did originally that we backtracked or backpedaled on related to subnets in pieces was we used that one BPC for backing services and one BPC for our Kubernetes clusters.
And then we would hear those.
And that would allow us to share that VPC for the banking services for multiple BPC is for different Kubernetes clusters.
But we decided against that because it just obliterated our Siders are available eyepiece because every time you divide by 2.
So we don't do that anymore.
Now we just run one large shared VPC with both banking services and Kubernetes clusters and related security groups.
Any more specific questions.
So I can tailor the answer better for you.
No one knows it yet.
So where do the split four counts is not based on architecture here.
So like for example without the request a block from operations center, or splits from that based on one side.
Yeah So different products may have different needs.
Wait a minute.
You need them art.
Yes because we're stored or data elsewhere.
So just really kind of planning your own it.
Are you guys managing subnet allocations then because you're basically you're saying it spans it spends what you can do an Amazon, it goes to other clouds goes your data center.
It goes to other places.
Yeah in this area.
I bet maybe somebody else might have some insights as well.
How do they mean.
How do you manage subnet allocations at scale for a larger organization that span multiple clouds.
That goes beyond kind of so Terraform is great from a practitioner perspective.
But how do you do it from like from a management perspective.
Your Craddock perspective, have you seen any software to manage these.
I guess a lot of companies just use ticketing systems.
I think it or hard like an Excel sheet.
Anything Yeah Yeah, I'd love to know if their software that actually does that.
No that doesn't go well.
Yeah, I can't get back to you on that.
Oh, yeah, they've been cool.
Yeah Yeah, I mean, I actually don't really know what's in it, but I mean, we run like nine data centers on our own.
And I know that they're sharing some nights between these data centers.
So Yeah, I think I like my wager is that they still use Excel spreadsheets.
But they're intuitive.
Maybe but I can ask.
You were not forgotten.
I just want to get through questions first, and then we'll fill it in with technology radar.
Sure take it.
Any other questions Terraform or IBS related.
I have a question.
But I've been having issues getting to the API because it's a private hosting zone.
Now I don't know what else to do.
It have to instantly be in a server internally.
So I could get my exile URLs to work or do you have an idea of what to do with that in the golf cloud account.
Man I wish I could help you.
We've done nothing with Governor cloud.
However, there's a handful of people in suite ops that do work with Governor cloud.
One of them is the usual is a regular here.
But he's not on the call right now.
So I'm guessing he's that reinvents or something.
So you know I've used gov.
Bobby OK fire away.
So my assumption here is that you're trying to get round 53 to work within gov. cloud.
So within the partitioned region of gov. cloud there is no about 53 service because it wouldn't make sense to have a public partition that is actually private because everything and need to be as Governor cloud as private.
So I mean, there's two notions of zones in rap 53 public and private.
And this would be like a third's you know state.
So they just don't offer it.
So you need a provider for your root account or a different sub account from your public a to bs account hierarchy the organization, your obvious Governor cloud account is parenthood by a public.
Yes or right.
So it's within that tree in on the public side that you need to allocate an account.
And then create route 53 resources that match up with your database Governor cloud resources.
Interesting one other question just because I've seen it mentioned by others in the committee are you working specifically with cops by any chance or are you using guess.
Yes cops because I said that I really won't go.
Exactly So yeah.
So do we eat with what do you call it with us.
There is a mode where it uses gossip to discover the nodes and are using gossip mode.
Yes And it's still not working for you.
OK So it creates the cluster.
But I can't get to the mast and no with the DNS provisioning internally.
I still can get to the master.
No What about with just the raw IP address of the master node that you would see through the console or otherwise.
So yeah, I'm not going to be the best answer that one as he cares or whatever is being used there try to call out the rule 33 each year.
It's actually not really.
I really wonder if I misspoke.
I meant the cops or whatever.
Yes, you know with the cops command you could just use you can attest to DNS zone, which is probably internal in it because it's gusset the base.
I'm just going to use the chaos and chaos.
But even with that.
If I tried to log in from a bastion I still can get to it.
I can who is private that would know I'll let me see if I can find somebody on Sui ops that would know the best chances.
Yeah Let's see here, we have a copy channel.
I think we have some I haven't tried to do what you're asking to do.
I did notice that the distinction between route 53 not being an in region resource definitely made using Terraform harder exactly.
Yeah, I tried it with my Camacho lobbyist count.
It was easy.
It was like a breeze.
In general, a lot of tools don't support gov. cloud as you've probably figured out right.
It's a region that isn't normally like part of people's suite of continuous integration tests.
So it never gets tested and never gets support that will allow you to beat your unity in the crops channels.
Is the bastion point to point to the right DNS yes and you can look it up manually.
Yeah So lately being is actually is within private also zone.
So any other machine within the VPC in that coastal zone can resolve.
So maybe the answer is you only can use VPN and your cops cluster.
If I am less of it.
I'm not going to use cops do you.
Do you have any idea ideas.
If I don't want to use comms for I believe gravitation journal has AI mean, there are other commercial distributions of Kubernetes.
Gravitational has one that is air gapped.
And I believe works with I think 3s also has air gapped install Yeah.
Gravity the gravitational they just raised a bunch of money to another 30 moons.
That's the business to be in.
Sorry I can't help more.
Well, it raises 25 million.
And I'm not sure because I didn't have experience with governmental zone.
But probably teleport is what you need to access from outside.
Does a cluster.
Well, that's an I think that's a optimization you can't be connected period right now internally using the DNS.
I believe when I use gossip mode with cops it was public cloud and it wasn't this.
And I was able just to use the IP at its all right.
Any So if I hear anything, I'll let you know.
I also reached out in the general channel to see if anybody can help me.
I'll be on the lookout for Google and do post back to you.
I figured it out in the end.
So we keep the knowledge helps.
OK Thank you.
Any other questions.
Well, you were when you.
Well, let me do a quick intro.
So as we were sharing here we're going to introduce the ThoughtWorks technology radar.
If you haven't seen that already.
Check that out.
That's my tab.
It's taking forever to load here.
Yeah, so if it's top share.
I can continue.
So the thought we're technology radar has been going on.
Well ego is going to give an introduction to that period is on the cloud posse team.
And let's treat this as a way to open up the conversation about cool things that they.
A link that Eric provided on his slides is outdated because a few weeks ago, a new technology out there was released.
It's evolving 21 issue.
So I guess you heard about technology radar.
But anyway, I will do a shortly introduction to everyone on the same page.
So the culture 80 is area apart.
It is regulatory.
So from SolidWorks is a company that do software development consulting and.
So they made a report about their opinion where technology goes and what is interesting what.
They have a good experience with what they are looking for and what they suggest to stop using and do it.
And a cool thing why I like it.
And follow is that they are doing this report for 10 years.
And we can look into the past and see what ideas and insights became a mainstream and what was the mistake they doing so report includes a wide spread of technologies.
And today, I will run through on the things that related to drops.
So escape our and light it from 10 machine learning and et cetera.
We is eager to allow your problems make a list of things we look at.
No, I agree.
I was cutting out everything OK.
Is it better.
So for one moment I can last 20 seconds.
So assuming which I like about this report is that when I read in it.
I found that there is a sense, we also do in cloud.
And this is like the whole report is, at least Kay notes to discuss locate and research.
So I hope that you will share your thoughts about different points.
OK so a report can assist f quarters techniques, tools platforms and language and frameworks and today we will quickly run through techniques and feel free to look in nasr quarters. to see if there is some tools you or like articles small notes if you have experience ways or you have any minds and et cetera and, you are welcome to share.
It will be very interesting.
So if you will go to technique.
So rather consist of focus groups in nature.
It is so adopt is a list of things they have a good experience with and suggest to adopting most of the project.
So trial is a list of techniques and tools platforms et cetera that have a good experience ways, but they're still no about the disadvantages and it doesn't fit for most projects.
So assets is as in they are looking for SSSS s that looks promising and hold is a section of a where they provide the list, you should stop doing.
Because if it doesn't looks, as a best practice. so go in to technology.
Section And what what. we use and do in cloud was different way.
So the first one is container security scanning. so really it's now a positive Docker Hub.
And Amazon Container Registry.
And Eric just mentioned before, the call about Aurora.
So kwe has been open source that was harder as it.
This is now ubiquitous in container registries.
So that's is it to adopt and start to use in India everyday projects.
This is it.
So this is a technocrat Dockery which can to find a no one security issues.
And if it's safe say make a mark and you can say so.
Another thing is pipelines for infrastructure as a code we so we have.
So we don't apply changes infrastructure needs.
So I see the, tools like Travis, called Fresh Jenkins et cetera.
We have Atlantis which is two that, follow like around tasks on Paul request opened on to her.
You should have but tasks that perform an actual changes are described as a code and stored in our repository.
So from that point of view, it follows.
This is this pattern for at least a year.
And, we can, and the mindset is this is a good practice.
And it is very useful.
And Atlantis is a good tool for such purpose of tasks probably better than most sites today because it gives you an ability to check what changes he will apply to sex.
Nicole and another thing, which is interesting.
And we are trying to adopt it is running costs.
There's a, architecture a fitness function.
So an idea is that, you should monitor the cost that the whole system and different some system goes to the level of port how much it costs you.
What they add to this is just that it didn't stand out to me until right now and I'm reading at the time.
So we've been using cute cost, which is open source open, core kind of communities cost tool behind the scenes and uses Prometheus.
It also works in California.
What's interesting here is what this is pointing out is that you can observe the cost of running services against the value you deliver.
So this gets kind of interesting where if you in Prometheus you have accessibility to bottom line numbers and your business order is sold or you know sign UPS or things like that.
This gets interesting because you can now back that out into what it cost to operate it.
And I think that's what the fitness function here is referring to.
Yeah but so so is this metric consists of two parts a cost of it cost metric and a value metric.
So how to collect fair value metric is business specific with all sorts of previous is just a database like synapse.
So basically, there needs to be Etl or real time you know it basically Prometheus exporter that that ingests that data from whatever source you have.
And then you can have, then you can truly achieve what they're describing here.
The question for me is how to calculate a value of a bitch sir at which point for example.
So cube cost can calculate he can show you how much each service you are running on carbonates for example, or plot cost you another tools that provides such type of information is spotting it, which is like a service, not yet the sales that you can cost, but it do a great trip once it does, but it's hard, but with spot is you couldn't factor in your own your own metrics as part of that, it will show you how much your namespace is in COGS costs.
So in this case, let's say, for example, for a period of time you sold a million.
And for that same period of time your infrastructure cost you know $100,000 operate.
Now you know that you have a 10x return on for every dollar that you're spending on infrastructure is it really an interesting metric in terms of what companies spend on marketing it for example.
I mean, I mean.
OK I don't know.
I mean, for us, I would not care about those architecture across metric.
I know I pays this much a month does is how much your customers.
I get each month month.
But yeah if I look at our customer acquisition costs that we calculate based on our marketing.
OK Each customer costs us roughly like of euros in marketing.
Yeah So yeah.
So ways it can be interesting.
And we have a Sas product basically, what is your offer.
What is your opex per customer.
And where that gets more interesting is perhaps their services that you're operating that have a very high optics.
But low but relative to the value that you're providing, for example.
So I don't I think it's up to the business ultimately decide.
I think one of the things that's been frustrating for me in this position with infrastructure is that it's always seen as a cost center right.
It's like where money's going.
But you're not showing value.
But you need.
We need we need to get better about showing the value that we're providing as well.
And tying that back out to metrics that the business uses.
So what are those.
I don't know.
Right Yeah points up here.
Let's what are some other interesting ones.
And those are things that we can talk about is a design system and days to provide a collection of design patterns of different components libraries and et cetera.
So this pops a future development et cetera and it slopes where familiar to what can we do with reference architecture.
That today, Eric showed on answering the question about subnets and end to what we do with these Terraform models.
So this is true for models is a collection of components.
You can like that interact with each other as it may go to make a really does anyone have something similar things that companies like documentation example, I don't know.
Well, let's go present.
So another interesting thing is that this is a binder at the station.
This is an instrument.
They provide a list of them, such as in total.
And Docker notary.
That makes you to make a encryption verification of binary images as ads that authorized for deployment and integrity clear et cetera.
So we had an experience with set yet in a bad way.
So we had a single storage for artifacts of photography images and that gives us guarantee we're on binary.
The same images across all accounts.
And if so, I said the checks and images.
Then it is.
It is like approved.
So by now at the station.
In addition to a to z for flow.
Is anybody practicing this here.
I'd love to basically.
I wish you know who I am.
Yeah, I honestly, I I've been trying to figure out how to write up my story without getting into trouble because it would be hugely embarrassing for a number of companies.
Honestly, the fedora people do this door.
Does this a lot of the package.
A lot of the official package registries do that.
But at the Doctor level deploying you're deploying cryptographic sign Docker images is one of the things Kind of here.
We know we should be doing it, but I don't really know anybody who is doing it.
Londo is a londo does it work with there or at least they do it for their Cooper daddies operator.
Like there they have some really wonderful Python framework.
Again, I'm kind of a Python guy.
I don't really know much about go.
But they have that baked in.
In fact, even just to contribute patches you have to have to register your p with an OH Yeah.
I mean, that's Yeah, that's the upside, though, right.
Well, the thing is that the operator is an image.
In other words.
It's the art.
You basically your artifacts become so yes.
Yes, it would be.
It would be on get upside right.
But they also sign the images they get built, because they publish those things out.
I mean, that's how that's how the code runs is as a Docker image.
Yeah Suzanne and Lynn do that to us.
Oh, yeah, sure.
A lot of sees.
Yeah, it's a funny the name of the repo is a little bit strange.
It's like one of their incubator.
Hang on a second.
I had it right here.
I put it in the kids in the office hours.
OK I sent you.
Well, I will check it later.
Yeah, that's interesting. really interesting.
So there is a dependency to do fitness function.
So an idea is to define as a metric.
How much dependency yourself, do I have.
And see if this is this metric measuring go up or down.
And then they control about complexity based on this function, we didn't have it.
Yeah, looks interesting.
It should be easy to implement and crosswords and kind of give observer but observer abilities needs to be changed at which point using an outdated original component or co-sponsored with the drift fitness function is a technique, which is specific evolutionary architecture for its functions to track these dependencies over time work needed.
Is this basically being able to see at a glance how out of date all your stuff is like your Docker images your home releases your packages your stairs relative to all of this et cetera.
Or I don't know.
I think the goal is that when you add something into your self to is you can see how much the dependency adds.
So for example, you don't want to read the Python tools integer days because it's take too much dependence.
And when it's not only you, but you have a like a La ti.
Yes, I did.
That's a metric that gives you what is going on in the project.
If anyone adds something to have a Yeah.
So there's that side, which is kind of as your 8 increasing the surface area of the code you manage.
And then there's also then the drift of all the dependencies in that as they get out of the tech that piles up.
So what I was working against with multi fire like because I mean, it's the idea is to be as up to date as possible and to track that how up how up to date are you dependencies.
That's how I understand it.
Yeah, this is where helm did or was at the helm Def comes in.
What was the modifier of.
I'll notify our needs.
Now a net exporter.
So we can implement the drift fitness function for home.
That is interesting, especially, like if he's got an upstream project that you don't know all that much about.
I mean, I was the I hung out with the Python sonic people.
It's a kind of an async web framework.
Lightweight and we're very nice guys.
Honestly, I was just there just really almost like you guys really just like my support group.
And I noticed that their package in FADARA was out of date and it wasn't building properly.
So I kind of like tried to.
I haven't finished yet.
But I've been updating it.
And they're all these weird changes going on in Python with Python 3 and phasing out a lot of Python 2 stuff.
And then all these weird web sockets things.
And it was like, oh, this could get really gnarly and Fedora like they want to use all the latest stuff.
But they still want to support the new things.
And it's like, oh my god, I think the Python 3 Python 2 thing would have exploded any function.
Well, it was interesting there is also like then what Perl has done with Perl 6.
Now They just renamed or I believe they at least voted to rename and I forget the name they've chosen.
But like Perl 6 was like a totally different language.
Let's not pretend that there's an upgrade path between Perl 5 and 6.
So they just they cut bait and created a new language that's like something for like 15 years.
I was exactly like the butt of a joke.
When I was in college two year in 1999 or whatever I remember talk of Perl 6 back then.
Bringing back memories, man.
OK So are you being serious.
In this section, there is it too.
Two points are related to each other and where we can go further.
And there is an interest in it.
So this is a security policy as a code and sidecar is for endpoint security.
Well, when we talk about secrets a policy is a codon our everyday practice.
We usually talk about IAM policies that gives it permission to some applications where we had grenades interact with Amazon and get taxes somewhere and plus a security submits and all that stuff said that manage security.
But here I found it seems that looks interesting to where we can read we can do more with this.
So this is an open policy agent, which is a tool that gives your ability to define different security policies as a code and it integrates with a lot of plot platforms and mesh services like is still on second.
Here is that this correction.
So it supports and why.
Kafka I don't know what that says.
This is the company.
And it looks like an incubator program.
Crowd native foundation.
So it seriously looks promising and really there is a two left to.
Like two fields where.
A lot of new tools. is this is a security end and cost control in a cloud.
So this one, if I have an instrument.
You should look at.
And I related to it is a side.
Carson as at end point security.
So when you use a public cloud and, you you're like you're on services across outside the bomb cloud provider then, you can use the comma sidecar as an endpoint security. and policy agent would gives us stand out.
How you can define a policy across different clouds and environments.
So has anyone experience with something similar.
Oh, how you solve a problem with security cross when you use a public cloud or something like that.
Because we I'm on Amazon on me and me.
They didn't are on public cloud yet.
You see all these demos for the service measures and to show how easy it is then to do the cross cloud networking using the match.
It's all on the flight path.
This is looking up apparently aside cars are now a first class citizen.
I think one of the ideas is it one that Senator Yeah you.
So I know, in the open ship scene there's a lot of discussion around.
Well, I should.
So Argo I guess has is getting some traction among open ship users and the first time I heard about this stuff was just a few days ago where people were asking whether they could use a p.a. to federate.
The author is the authentic the authorization for Argo because the thing about Argo is it's got like its own.
You have to off into Argo and you also have to offer into shift.
And so it's a little bit and an hour goes like really, really not.
It uses decks.
So it's very simple to integrate like Google OAuth or get lap or GitHub super, super simple, but it's still like it's own thing.
So it would be nice to have it somehow federate so that would probably be like maybe in those like CCD apps where you're running like nested administrative domains or maybe that would be a place to do that where you could run.
You could say you know you only have to.
Off into like the parent container.
So to speak, or the parent framework, a little bit like what AWS tries to do with cognate ho me.
So I still kind of what you're saying.
There's a need for something in Kubernetes land which makes it easier to standardize authentication across apps.
Yes and make it all work.
And I agree.
And we've spent a lot on this.
We personally use key quote, but I mean, it's just like everything's just a hack on top of the hack on top of a hack.
And then we keep low.
We use gatekeepers and we deploy gatekeepers for every single service we want to expose.
But even though we do that, all we're doing is providing authenticated way to access the app of which very few apps actually then handle.
Yeah, fine grained access controls the Cuban dashboard is the exception to that where we can actually pass through the roll, and then the grenade dashboard honors that.
But yeah which is more standardized what.
This is the challenge, though, with open source right.
Integrating dozens of technologies, from Def dozens of different vendors of some use Kubernetes and some use something else.
And I'm not holding my breath.
Well, this is what Opie is trying to achieve.
Right I mean, look, I think technically what they are trying to solve and they're doing a great job.
I mean, I know is up to German bond or Dutch bond is implementing their this gig compliance c stuff, which is now and they're pretty successful with it and lots of it's getting a lot of traction in Berlin right now.
That's from what I see.
So it works.
It on boys so that I can add on boys a proxy.
So I see that connection point there.
Then let me just put a link in the home office Horace.
OK, cool it degrades with a lot of things.
I mean, I open it at least.
Well, then that is encouraging.
So that's all from this section.
If you like this far.
I would be happy to continue.
Next problem is to set platforms especially we have experience with some tools and platforms that mentioned set up, for example teleport dependability et cetera and.
And if you choose something that you use.
We can and we can discuss it.
So that is center a much.
And your thank you.
Yeah, thank you for bringing that up.
Well, I think we'll be doing is we'll be pulling some of these topics out and adding them as key points for future office hours as well.
So any other questions, guys be totally unrelated meeting a talking point on things as well.
Yeah Yeah Yeah.
Cool Yeah, totally totally.
It's the elephant in the room right now.
Is it obvious reinvent.
So what are some cool announcements there.
Out of the 150 announcements or whatever that they're making.
I feel like it's almost abusive at this point, the number of announcements that they drop on us in one week.
Yeah, I tend to just wait until it's done.
And then start watching them on YouTube.
So I can actually.
I mean, there's lots of means meme going on Twitter right now.
We could go racing or we need new lines.
We just put like a coat check that this bill line is code per line of code.
It's all like, oh, yeah we'll need new lines.
We just put it all on one line and don't pay anything for the servers.
So related to reinvent but I got I chatted with me of US rep.
Yesterday about the new savings plan stuff.
If you haven't had a chance to look at that.
It's well worth your time.
Basically, it takes the best part of convertible reserve instances and ends rather than having to commit to certain family in numbers of instances in normalized units and do all those calculations.
Especially if you're rapidly changing between the six types and scaling what.
That can be really hard to figure out.
You just commit to $1 per hour amount and it applies across all your compute and cross region, which our eyes don't do.
Our eyes are always region specific.
You can never convert though.
So it's actually pretty cool if you.
Now as far as I know, they announce beginning in November sometime.
So definitely I would highly recommend looking at that.
Talk to your accountant make sure it makes for your use case.
But I do not see a use case where it's worse.
I haven't yet seen you use case where it's worse to go with the new savings plans for your e to compute rather than reserved instances.
Yeah So just keep in mind there.
When those start expiring.
Yeah, that's interesting also.
When if your eyes are expiring this is probably the way to go after that.
And there is the savings of that relative to using spot for example.
And how big the delta is between.
All right pricing and spot instances.
So the savings plan pricing for me.
I was looking at partial upfront three year dollar commitment amounts.
And it was roughly it was right.
It was just about 50% off your high demand cost.
Nice So pretty nice.
The obviously, it's pretty comparable to our eyes.
Some places I saw was more percentage of the on demand and it was never below that I saw.
So as I said, I don't see an instance where it's worse to use to gain the flexibility of the savings plan stuff.
They only have four easy to compute, you can't use our eyes for art.
Yes, they have a lot harder job tackling RDX because they have to deal with software licensing for SQL and Oracle and all that crap.
And then they also have they don't have it for like a to cash instances yet either like they have our eyes for it.
So those you still have to go through our eyes.
But the savings plan.
Now I hope they announce something it RDS or reinvent or they're just like, yeah.
We're going to expand this to RDS and the massive cash this year because I'd rather just say, yeah, I'm going to spend 30 grand a month on my compute m 20 grand.
RDX please give me a discount on that guaranteed spend and go from there.
Yeah, they bring that up.
I forgot to bring up the thing about savings plans.
That's in pretty good cost optimization.
Trick that they introduced just jumping back and.
I'm pretty sure we'll be talking more about reinvent next week, since we're already up on at the end of the hour here.
Any other big announcements obviously casts far dating a big one.
And then the week before last just previously last week, whatever was the fully managed node pulls for you casts both have Terraform support already, which is cool, because that means that Amazon is working directly with Tashfeen corp to get this stuff ready for all announcements to fill in.
Thanks so much.
I don't know if that was a joke and fuku and CloudFormation.
But yeah sometimes even CloudFormation lags behind right.
So be interesting terraforming usually nice sometimes.
Usually that is interesting.
Yeah, let me know, if any of you find that out posted back in office hours it'd be interesting to find out.
I don't know if he or.
Actually, this is if any of you guys want to monkey around with like cut some of that cost stuff.
I have certainly for the next couple of months.
I have pretty much unsupervised access to a bunch of demo servers and resources in similar logic.
So we can do all sorts of aggregation of logs and metrics and bye.
Thanks for extending the offer blades.
You guys can hit plays up on the officers too.
Yeah, just exactly.
Yeah All right, well, then we.
That brings us to the end of the hour.
And that wraps things up.
Thanks, everyone for sharing ego, especially for taking time to prepare the notes on the technology radar.
I always learn so much from these calls and a recording of this call is going to be posted in the office hours channel.
See you guys next week.
Same place same time, guys too.