Public “Office Hours” (2020-01-15)

Erik OstermanOffice Hours

1 min read

Here's the recording from our DevOps “Office Hours” session on 2020-01-15.

We hold public “Office Hours” every Wednesday at 11:30am PST to answer questions on all things DevOps/Terraform/Kubernetes/CICD related.

These “lunch & learn” style sessions are totally free and really just an opportunity to talk shop, ask questions and get answers.

Register here:

Basically, these sessions are an opportunity to get a free weekly consultation with Cloud Posse where you can literally “ask me anything” (AMA). Since we're all engineers, this also helps us better understand the challenges our users have so we can better focus on solving the real problems you have and address the problems/gaps in our tools.

Machine Generated Transcript

There we go.

All right, everyone.

Let's get the show started.

Welcome to Office hours.

It's January 15th.

My name is Eric Osterman and I'll be leading the conversation.

I'm the CEO and founder of cloud posse.

We are a DevOps accelerator.

We help startups own their infrastructure in record time.

By building it for you and then showing you the ropes.

For those of you new to the call the format of this call is very informal.

My goal is to get your questions answered.

Feel free to unleash yourself at any time if you want to jump in and participate.

We host these calls every week will automatically post a video recording of this session to the office hours channel as well as follow up with an email.

So you can share it with your team.

If you want to share something in private.

Just ask.

And we can temporarily suspend the recording.

With that said, let's kick it off this Office Hours.

So here's some talking points.

I put together one announcement I'd like to make.

Just today.

I think hashi corp. announced that the Terraform providers are going to be distributed as part of the Hasse corp registry.

So this is like what is available today with Terraform models.

The caveat here is right now, it's only official providers they are working to open it up to third parties and other providers to publish their providers.

I guess for the security implications of this.

They're just being very careful about how they go about that.

Anyways, this exciting news because anyone who's tried to use third party providers.

It's been a real pain.

So if in the near future, they fixed that I'll be very happy.

Another public service announcement.

We said this last week.

Just want to share it again, because I'm pretty sure it's a bunch of companies are going to get bit by it.

The root certificates that effect RDS are raw and documents are expiring in March.

If you're using certificate validation with your instances that's going to break unless you improve upgrade your certificate bundle for those instances or fruit for the nodes that your clients run on those certificate bundles need to be updated.

All right.

So the next thing we have a question from here in our community posted and office hours.

He has also some feedback because he asks in Cuba and Eddie's office hours and got some answers there too.

So we'll see how that reconciles with what we recommend.

And then a demo.

If time permitting, we wanted to do this last week we ran out of time.

I just want to show you how we've been able to use CloudFlare as kind of like this Swiss army knife reverse proxy port injecting H2 MLC ss into sites that we don't necessarily control.

All right, let's start with you here.

So the question you'd asked was how to provide a module or a tool to our application developers to provision the databases for them, including the deployment of the secrets inside of communities and in office hours hours right now in that channel.

If you check.

There's a link there to a Terraform repository where it looks like you started kind of like EPBC with putting something together.

Well, it's basically for me, it's in our infrastructure as code repositories right now.

And we want it to or would like to move SAP basically to the application developers.

This is a private project of my the similar is pretty similar to the infrastructure that we use at work.

So you would go to the committee you just have modifier telephone from file.

Yeah, this is Bono.

All right.

That's another one for post.

I think Costco's Hamilton fire down on the bottom.

So easily these two would be one file and we could give this to our application developers.

And I can just maintain those and currently we get roughly around 2 to 10 requests for requests on our infrastructure scope repository and make some people.

And we still have our own people.

So currently kind of gets to the point where it's cumbersome to maintain all these requests and we were thinking about just like putting it out.

So I asked the same question in the communities office hours where a lot of knowledgeable people on.

They basically said, well versed in their developing experience because the environment is either.

So I don't know much about op's application therapy in general.

And yeah it just reduce their productivity from the application developers, because I didn't know the centerfold.

That's the feedback that I got from them.

But I'm happy to know what if you are doing something somebody.

So there's two things at stake here.

There's one is kind of like your stated approach or objective that you're trying to do right now.

And this is one way to do it.

What you describe here is kind of like how I would automate the production of perhaps your production grade backing services for how low to fire for those of you who don't know how notify or is a project that year working on.

It's a helm.

It's a tool to visualize differences between helm releases or helm charts.

And here is kind of like some of the infrastructure code to deploy that.

So what we have here is some Terraform code that provision that generates some random secrets that provisions a post gross role in the database, and then writes those secrets that were generated into Kubernetes into a specific namespace with those values, including the database, you or I in this case is what he's writing there.

So hold that thought doing this and Terraform is great.

And at some point, you're probably going to want to do that like for your production grade services.

This is what you're going to do.

Well, we don't recommend though, is for preview environments or pull requests environments or what we sometimes call unlimited staging environments.

We want to use Terraform in this case in this case.

Well, we would be using is as part of your helm chart you would have the ability to enable backing services that that chart depends on.

And then that makes it very easy to spin up a fully self-contained environment within Kubernetes just by deploying that chart into a release that.

Then the chart itself can generate those random passwords using the Rand function, whatever it is in home templates and it's easy.

OK So that solves kind of like I think the problem actually that you're trying to address right now.

But it doesn't solve kind of your initial requests like, how is this the right approach that we should take for safe production environments.

If I reframe your question.

And so that was the way I first, they actually interpreted it when I was senior question and it's something that we've been thinking about to a cloud passes like, how can we move the dependencies for backing services of your applications that we need to provision with Terraform closer to the applications themselves and not in like some centralized infrastructure repository.

Because you don't want to have to open one PR in the infrastructure repository like most companies have.

And then another PR free application coordinate that whole process.

Right the main problem because we only on regulatory push our infrastructure is cut to production or even to staging and application developers might be faster than us and want to stay up as a service to staging already.

And they have to wait for SRV to dodge a gift with a message.

That's a good point.

One that I didn't bring up.

So you're also talking about not existing cert not just existing services, but also net new services that are deployed.

What's that process.

Yeah OK.

So be the approach that we are starting to lean towards.

But I can't say that we have fully achieved it.

I think there's some others here on the call that have maybe Dale has done it or Andrew Roth maybe if he's on.

But would be to have the Terraform in each one of your microservices.

You have a Terraform application folder in there and that would have the deep.

Backing services that it needs.

Now what you describe here.

Yes, this is a perfect application of what you just what you have in here.

This should be captured inside of a Terraform module.

And then that module will be version separately and of itself.

And then you might want to have some automated tests related to that separately.

But then when you want to instantiate that.

How do you do that.

Well, that's where you're now in your application full.

Now in your applications repository let's say how modifiers your application repository there'll be a sub folder in there called Terraform and that's where you're going to invoke that module as part of your ICD pipeline.

The question is then what do we do about the integration secrets.

How do we get those there.

So that's going to work well in this case is that depending on what you're using for your CI/CD I'm going to say, let's say you're using Jenkins if you're using Jenkins it's going to use the standard credential store and then your pipeline will indicate that that's the credential store, it's going to use.

And then it can just go ahead and position this stuff from scratch.

But the challenge is if you have some credentials that can be programmatically generated and some that needs to be user generated.

That's the challenge.

And that's very often the case like you're integrating with third party guys and you developing a new service.

How do you get those secrets in there for the first time.

I don't have a brilliant answer for that.

We usually just see people kiddos in as part of that cold start.

Well, this kind of stuff your itself with time from very able.

We just put them in well.

And just put that very terrible variable in top secret configuration.

So that's how we do this kind of stuff that's almost an.

And then that also empowers the developer to set that environment variable in your CI/CD.

OK Yeah.

So this is how we do.

We are doing that or they are using basically the services and just to jq and try to get it somewhere else.

For example, Google Maps.

Someone wrote a best script to generate Google Maps secrets.

OK I don't have Terraform support.

But yeah if someone has ideas about us.

I would be super happy to know how you are doing this kind of stuff if you're doing it.

Mm-hmm It's more like exploratory right now.

So also I'm curious.

I mean, anybody interject in the answer I gave is was there some disconnect I missed there, which is the part that you're struggling with.

It's more like I wanted to know if people are already doing it and what our experience was with it because, well, that's the feedback that I got from the committee's office hours.

Yeah, I tried it.

And developers didn't understand why I'd now have to approve infrastructure as code as part of their deployment process.

So I had to prove they did it this way.

They approved basically the terrorism plan and the terror from Kenya to execute it.

And then the application promotion got executed.

And larger.

So you doubled your approval go gates because they moved to infrastructures called into I think it's going to largely that the organization culturally how they approach it.

I kind of react to that.

I this is this kind of like, oh keep other people's problems as it relates to code reviews.

I kind of reflect that.

Don't you want to grow you know like career wise and learn these things.

And I think requiring code owners in your repositories so that certain when certain files are changed that an approval comes from one of those people.

I think that's a way to mitigate some of that.

But that can be all bundled into the same code review process.

Perhaps it slows it down a little bit initially.

One other thing I wanted to say is.

So the challenge here is that we're spanning tool sets.

Right like if all of this works just through coo cuddle.

So to say or like the Kubernetes API it be a little bit easier versus here where I'm sorry.

Go ahead.

I'm saying long held versus here what we're doing is we're straddling tool sets we're straddling a Kubernetes tool set and Terraform toolset for deployment.

And this is where I think that now.

Now, if we look in this repository you're technically using Digital Ocean so it's a little different here.

Yeah And some of our answers would be probably more obvious focused.

But this is where the e.w. s service operator is also, I think, really exciting is the ability not necessarily this project.

But this technique.

This badges.

So you have a Cuban ID.

Oh, interesting.

It's been archived.

But basically, you have a CRT that you deploy that defines that part of the infrastructure.

So that you can stick within the same toolset to deploy.

That's cool.

I actually today found cube farm, which basically gives Cuba need to see ID for telephone providers.

I posted in the office also.

Oh, I will not exploit get net but I think it could also off my issue where I said, it's a second link not the first one where people now see I want to switch, or maybe it's the same coupon coupon looks like this is they've gotten a lot more in.

Yeah, I guess I have not seen this one because ranchers started with one as well.

You know.

Yeah Yeah.

I have not tried it out yet, but it seems like you get a C see d for your telephone providers and can then do the same stuff.

Mm-hmm Yeah Q4, they also do this.

Let me double check the OK.

So it is just I thought Jim form did more than Terraform but OK.

Never tried it yet.

So I. Just want it today like a couple of hours ago.

But I will take a look into it as anybody kick the tires and in Q4.

Yeah nice.

That is the first one she heard about it.

Yeah or it was this based from the one from Capgemini or is that a for I'm not sure.

I did not investigate too much, to be honest.

Other products as well.

Cute folks keep farmers from like app code.

Yeah Yeah.

That's what it was.

It only went to the website saying absolute they do a handful of things I haven't looked them too deeply.

Interesting looking products.

Yeah, thanks for sharing.

Yeah you nailed it.

That's kind of like what I was thinking is.

I think that's where things are going to get easier and easier.

There's more of this is that we can use communities as the scheduler for everything to do.

Cool well I didn't want to embarrass Dale here a little bit.

Dale just got his certified Cuban his administrator exam K And so pass that.

Congratulations Dale.

Glad to.

Glad you accomplished that.

Great Thanks.

And that means anybody else is serious Cooper related questions, ask Dale any other questions see here got something that chat that someone actually do the Cuban writers application developer certificate.

Oh the.

Now I Katie Yeah.

Is that next on your list.

This one actually came about a little bit.

Maybe I think to go more into the security side of things as well.

Data science.

I'm not sure.

I may be doing them watching.

Yeah by the way Dale posted the resources he used to pass that exam in the humanities channel in suite ops.

Check that out.

Yeah, the first link for Linux Academy really does it good.

Or if you want to brace yourself itself, but the one on you dummy itself actually does this practice test that are pretty spot armed with a set of questions that actually go with it.

So are doing both of them you pretty much cover everything.

I have a question does Michael seem Michael a long time listener.

It looks cold wherever you're at in New York City.

I finally kind of gotten the weeds with this product.

I'm talking about was just started here, which is the rule, you are all service within a code fresh and so you know one of the things I do is I was to create a branch context read your book to create a project.

The project has its share configuration.

So acts like a meadow namespace and then what I'm doing is because we have essentially microservices that need to as an application must not support.

So what I've done is I've used Helen file right to basically install all of the applications as they are from the release branch of the latest basically, from our sort of the next branch of the charts.

Then basically running it again with just a diff image tag.

What have you.

I'm trying to figure out the best way to do that next part do I. I tried to do help while apply with just the you using a selector and with a namespace.

But that would seem to do just it all just ended up being reverting back to all having the same image tag and not like overwriting it with that diff.

I'm not sure how to make it a command line.

I have to write a file.

Unsure about what I needed to do in that circumstance if that makes any sense.

Yeah let me summarize then my own words also add some more context, I think for everyone else.

So see you're using ranchers distribution community rancher.

This concept of projects and you're using code fresh as UCS platform.

You want to create preview environments or like these unlimited staging environments as we call them.

And to do that.

You've defined your entire application architecture inform form as part of multiple helmed charts or releases and all of those charts the releases are in this health file.

And now you're deploying that file as part of your CI/CD pipeline.

And when you build a new image you kind of you want to update that service.

Now, the part or this is the part where I lost you a little bit is the problem like, why don't file apply wouldn't be working for you.

Well, I mean, I think I was running a health file sync at first while I biggest because I guess initially, I'm doing this in the model regrow itself.

Right like so it's not all of the list right now at the moment weren't being published because we still kind of build a pipeline.

So they weren't officially published.

We don't have a version.

I mean, no question when asked the versioning system for this.

I'm trying to figure that out to be able to create canary builds that kind of stuff.

And whatnot with you know I have other tools I use for my libraries that I obviously like NPM libraries that use like auto and semantic versioning, but unsure how to be able to make that work here.

Aside from that what site was a question you asked.

No, I didn't.

So I struggle to see where I didn't understand the problem you're talking about with drifts or patches or things like that.

Were you using tools outside of the home or outside of the home file to do the patching or changing images.

No I mean, you know what I mean.

I just use code fresh to create an image tag.

And then I don't like save that file.

I just kind of pass that as a set you assets as a set to the second.

Like yes I do the home file plan, which applies everything.

And I really do help upgrade with the diff.

You know for the one service that I'm trying to see against the overall and trying to.

I was unsure a what.

Which one to run like Darren helm file sync a second time he'll apply the first time I went.

I'm not entirely clear which I should be running in order to be able to your fly.

It gets to think.

Yeah Yeah.

Yeah, I understand the confusion and so part of the confusion is like hell fire as the tool has been growing really rapidly accepting a lot of cars and things like that.

So sink was the original command that helm file came out with.

But then to kind of more closely aligned with the pattern of Terraform where you have plan and apply and destroy the whole file implement that the same things.

So from what I recall I think helped file apply honours that installed flag inside of your releases so that you can maintain the state of weather services installed or not installed while helm files sink.

I don't believe we'll uninstall services based on that installed flag versus otherwise.

So that's that.

Now what you're doing.

I can't I can't say it's wrong.

Like the two phase apply process.

Well but it's different than what we do.

So what we do is that configuration like 4 4 held file, you would use environments.

For example, in the environments if you like.

You don't want to rewrite that file if you want that dynamic and that can just refer to a go template macro for interpolation function for get in right to get there.

That's exactly what I'm doing actually to be clear.

OK I was doing that itself.

And what I found was just weird stuff like music.

Initially I'd like I have environments like with ephemeral and broad and QA which is where my static aesthetic values live right.

So that's where like you know the main thing is that the image tags is what essentially, I wanted.

All right.

Now, if I tried to float up the end up everything there then everything would have the same very value that I set.

But that just sounds like the schema is wrong if you're held a file.

So like that.

And I'd like, OK.

So not everyone is familiar with how file Alma file is a tool that helps you describe all the helm chart releases that you want to have in your cluster.

And the problem with helm is right that the values file is static.

It doesn't support any templating any interpolation functions.

Anything like that.

So the tool help file came out.

Let you describe now what helm releases you want to have and lets you as a developer kind of define your own schema in the form of what they call environments.

The problem with helm and helm charts is that every Helm chart every developer has their own way of describing how to install this application.

So it looks like a mess when you try and stitch these things together versus the helm file.

Now you can define that schema.

All right.

So in the schema your case, you have somewhere where it says image tag.

But it sounds like you only to find that in one place.

It sounds like when you want to achieve this you want to define the image tag for each service.

In this modern reboot what does difference.

What I'm trying to do for the ephemeral right is not only define I have a default. So falls back to like the next tag.

Yeah which is what we tag everything with.

Once it goes away.

What then I want to override just a single image tag for a single service.

Now what I'm actually doing now because I couldn't figure out how to use it twice.

So I'm just doing the whole file sync on everything.

And then is running the helm itself.

I guess the one released and upgraded that I say.

So Yeah Yeah Yeah like that can work.

Yeah, I think maybe if you're able to perhaps after the call or even during the if we have time.

If you can share some portion of that home file that you have been better at answering.

Also the other day the other concern is that using the next tag while it makes a lot of sense to humans.

It's also really hard to really have assurances of what's running and what we would typically do is we.

So we always tag the images with multiple tags and code fresh makes that very easy.

So Yeah, we may be tagged with next.

But we always tagged with the commit shop and the commit shop.

I'm doing the commission with the code fresh branch normalized thing in front of it.

So yeah.

Which is what my name spaces while I also create a new namespace each thing.

So exact that the image tag is that.

And that's for the federal environment.

And then once it gets promoted to the next environment.

It has that plus the next day.

Yeah, I'm only doing next because it was like, how to like how other than pulling down the latest release from code fresh from the registry.

Once I do that, then it will be that I can use that instead.

So let's see here.

I don't know how I can quickly.

And I go upstairs will pick and go to the computer.

I'm on mobile right now.

OK Yeah.

I don't think of an example.

I can pull up.

Well, the reason we fast enough for this to not waste a lot of time on the call.

I don't mean to bother everybody's questions is probably go while I'm getting upstairs and changing my zoom to the computer.

Sure thing.

Also here.

Are you using helm file as part of your process or not.

Yeah, but not us.

Michael's planning it to do.

So we have basically each application has a helm shots.

And then we have basic infrastructure, which is deployed by hand file.

So basically, our set manage our ingress.

All that stuff was two part time file.

But the other stuff is not thinking about what he's trying to achieve.

I think he could use jq to see which tech is currently report deployed.

And then check basically, if there's a difference.

And then only deploy the same file apply the changes that are actually out there because I think and sync and apply.

Difference is only that there's.

So my understanding if we look at this example of that.

I add here is that you'll have many releases of related to this one monetary one.

One could be bold for example.

And then there's an image and then there's the tag here.

And then he has multiple releases.

So here's bolt and then down here will be another one like refiner.

And let's say he's reusing this syntax everywhere where he has the image required environment variable image tag.

But he has that repeated everywhere.

Then how do you target just want changing the image tag for 0.

Maybe I understand.

So that your problem is that it's not deploying maybe if because you're using the next tag and if you just target one service.

How do you just target that one service to update.

I think the disconnect here is that if you were using to commit shy instead.

And then deploying like refine it with that commit shot.

Then that one will update.

But not the other ones because what you can do is you can combine required and with like a coalesce so you can say like Grafana.

Image tag.

Otherwise default to the default image tag.

So that lets you override individual tags for each individual environment or individual releases.

But I am guessing he is in the staircase right now.

I will I will pause that temporarily.

I'm here now.

Sure can share that.

OK go to office hours.

Was that my thing or should I share my screen.

If we want to let this go.

Yeah, we can.

I'll start right.

I'd rather not you know share my code on a thing because.

OK if you don't mind, I'm sure, you can also just as a reminder, these are syndicated.

Yeah, that's OK.

I don't mind that.

I just you know.

So you see my screen now.

So Here's what I have now.

We have a model repo obviously.

Each of the services up here, and then I just a ploy folder in which case, I have the charts themselves.

Yeah And then I have the releases.

And then I have appetite.

Yeah Yeah.

This is kind of what I'm doing with this right now.

Yep this looks good.

Now show me where the image tag is to find the image tag right now itself would you define for a environment.

Image right now is here.

So define OK next.

And then I override this.

But there's also set in my view, in the chart itself, which is.

Yeah So it's just the that I would do would you want to do this replace line 16 with a go template interpolation.

And there, you're going to look for the value of articles image tag and if articles image tag is empty, then you default it to next.

I would get that by clear.

I'm just doing this right now.

So it would be.

Yeah articles knows the way articles.

No, no, no, no.

So put like required and not require them.

I'm drawing a blank on the exact function name right now.

I think it's getting this.

Or is this just.

OK And not.

And space like this right.

Exactly no.

Yeah, I would stand you know standard environment syntax.

So be or.

Exactly So like that.

Yeah But what I would actually do because you want to surgically target one of your releases not all of you.

So now what I would do is I would change that to be a prefix that with articles image I got it makes total sense.

And then whenever I'm going over something if all value is to be next.

OK So behind idea if you would basically do a pipe and then default next.

Yes, exactly right.

I gotta say.

So the default is here.

That pipe pipe default next is a pipe.

I thought it was the other one I think.

I planning for a pipe.

So sheer pipeline and a string with next guy makes it work.

Yes each of my services in that way, I can.

Exactly So now you can surgically target each one of these with the tag you want to deploy it, which would be in your case, probably the commit shop is right on.

Well, thank you.

I appreciate that.

That's an enormous help because I know that you would do that.

If I do this.

Now I can just run the helm file just once right because yeah you just run it once, then you call start to export or whatever, just set that environment right also.

But to test this, I would just like to just export a ticket image tag and to have high Def and you get a Def and a church only chose this.

Let it change.

Yes it's changed.

So I said export Oracle's image tag.

And then do that locally.

Yeah Yeah.

OK that actually would solve many of issues.

I'm Morgan Brennan to double double the point.

So thanks, guys.

I appreciate that.

So oh come on now stop my share.

But I appreciate it.

You know I was looking forward to this like all week sounds like you finally have it is you that's like hamstringing things that Secretary has a question regarding to get UPS if there's no one.

Yeah, one quick question.

Do I get rid of the initial values that are in charge themselves only use his values.

No you don't.

You can leave those.

It doesn't matter because values that gamble will override whatever you define in those charts.

So if you make your chart something deployable by default, I think that's generally the best practice.

And now what I was doing.

Also thank you.

Next question.

Yeah, I feel that if anyone has any like experiences with pipeline that's code managing with many multiple development teams.

I'm open to hear it.

And I'm at a point now where I'm at, 20, 30, 40 repos and know there's old pipelines code that I have to redo and I'm struggling just working with some offshore teams and they won't let me touch stuff.

They won't accept my peers.

And it's driving me out of my mind.

Well, I'm kind of curious like if anyone's strategically worked around that other than myself.

I'm working on like using outside libraries and some of my pipeline is code and you know you know I keep on thinking I just want to do my own dedicated branch to do that.

You know.

That just seems wrong to me, seems like an empty pattern.

So I'm curious if there's any other.

Experiences out there.

So I can go on talk about it if you want.

So I had this.

I had a similar issue.

We had outsourced one of our projects and the team was really afraid of.

If I do the pipeline stuff and they had a very weird schema.

I think I a similar issue right now, whereas I deployed to production into a branch and has three different benches for environments correct.

Yeah, we are multiple branches we're doing per environment branch at the moment.

So one of the issues that annoyed me the most was that I never knew which features were deployed in which branch.

And so I Justin also had what I was saying goes to mass and we use tick UPS to promote this was the only way for me to manage it.

But also you could have the deployment process in a separate repo and have just updating the array will come across or put up the available talk attacks pushed into a different repo out of the cope update.

And then manage the deployment process in this secondary repository.

OK That's why I'm starting as a viewer toward.

So at least I know I'm not feeling like I'm totally off my rocker and go on that route maybe.

Eric Yeah.

So if I also know that low end under wrath has some experience with this.

I think I might have misinterpreted the original question.

If you want to just restate the problem again.

Well, so I've got a lot of pipeline as code in the repos as it probably should be.

It's supposed to be, and I'm to the point now where I've got several many multiple repos that I'm helping manage as a singular DevOps resource amongst multiple teams and certain teams and projects are more sensitive and they will not allow me to make pipeline changes easily.

It's just seeming like for although I am in a release you know I'm going to try to maintain a cadence with the development teams in some cases.

I need to make changes to their pipelines to energize and improve things.

And then I'll let you know I guess like a better way of describing it.

And it seems like there might be a better way to do so.

OK So they're concerned about changes in flight to kind of these pipelines that could reduce the stability of deployments.

So don't blame them honestly, you know I don't know.

I mean, these things break relative you can break pretty frequently.

OK So one other one other kind of procedural thing or organizational thing is adding code owners right so that when changes to the pipelines happen members of their team that are set as code owners fought for those pipelines have to improve on those changes.

So that's one way to enforce it without disabling kind of like ability to do pipelines as code.

So that's one thing.

And then.

So pipelines.

Yeah, that those themselves have their own lifecycle almost right that you need to be able to validate that that's not breaking things and you have are you are you a what are the common types of changes to the pipelines.

And are those pipeline kit.

Can those changes be validated, but by using preview environments.

And that's the changes that I'm looking at.

So I mean, we put these things together about a year ago.

And there's a voluminous amount of pipeline delivery code and things like that that I personally wrote that I'm not even proud of.

But it was needed at the time right.

Yeah So you they're saying, you know and that is we all know the more could the have and your pipelines the less simple it is and less important is the more it's Yeah breakable.

Right Yeah.

So I mean, you know I'm so I'm trying to simplify things down and make it declarative as possible and going back to some of these old still working pipelines and trying to like remove unneeded elements and things wrong along those lines.

So I am unfortunately stuck in Azure DevOps in this particular case.

But I'll use it to its fullest to do things like pull out pipelines into its own repo and reference that repo externally.

And I greatly simplified a lot of the various repos pipeline code.

But in the same regards I'd point to an outside repo for the code.

So that's better or worse, that's what I'm doing.

But I want to do this more for some of the other for some of the more delicate pipelines and I just I'm just making them tripping up and working with some of the developers on these things and maybe it's just me.

And you know.

But I just wonder if anyone else has dealt with that in any other way.

I keep on thinking, maybe I should just have a dedicated branch for pipeline code you know in each repo that they can't mess with and that sort of strips the control from the team.

And that's not that don't feel right.

You know some way to break other things.

So I don't know.

That's a sort of deep coda.

Are you able or using get up or using another piece.

Yes it's the area of DevOps repositories.

Oh, OK.

I'm not familiar with those.

It is a lot of hassle oh in the audio environments.

OK Yeah.

So the code owners might not apply to that.

Right on.

Maybe you in their branch policies and whatnot that I've been slowly been able to get enforced.

But yeah.

Code owners is they've got something similar.

Yeah, people notify you when there's changes.

But then getting them to approve.

It's a different story.

So what maybe I can have a private conversation with you because it's maybe been over the scope for this conversation or for this talk right now.

But I had an accountant exactly the same issues that you had this application team removed the info as he pipeline stuff that I introduced.

And because I have like 50 to repositories that I'm working on.

Yeah, I wasn't aware of it.

And then they said, well, we are not able to put any more because I restricted the UCLA access.

And yeah.

I wasn't even aware that's a push differently now maybe.

So if you want to 52 plus repositories at the same time right.

Gets hot, especially if you have 120 injured.

Yes stuff.

All of really smart people you know it's not like you know.

Yeah, they'll find a way around things.

Yeah, I'd like to chat you on this.

I appreciate that.

Thank you.

All right.

It's really nice to hear from I I was going to wait to eat or was asking if we were still going to do the CloudFlare demo and I do want to do that.

I was going to give him a second to join here.

I'll get CloudFlare teed up in the meantime.

Any questions.

No, I love puzzles.

Well, I guess if we're just I'm trying to play GitLab on queries and it's going to be publicly facing.

So I'm trying to do it like I wanted to be an exemplary deployment and I've been using their help chart or I should say the instructions for their Helm chart.

And I'm thinking, wait a minute.

There's got to be a better way to do this.

They just seem to be lots and lots of little moving parts.

Like oh you wanted to s three.

Then you got to go over to this demo file and tweak that.

Well, you want to use those three with IAM roles.

Oh, hang on a second.

Now I run it.

So suddenly I find myself trying to configure k I am.

It's just it's not that everyone, or is it just me.

It's not just you.

OK I will.

You'll you'll see all these beautiful demos of my I did.

It's that easy.

But the reality is once you get to like operationalizing something for production, you've got to worry about emeralds.

You've got to get those roles into your services.

You gotta choose if it's like, OK, in this case, you need an S but it'll keep you've got a provision that lucky you got the back.

So that's why we have so much freaking Terraform stuff to support coober neti stuff because there's so much that Cooper needs can't do.

That is like cloud platform specific.

Yeah, this is where you know it's just that we're a little bit early in terms of where things are in humanities but like the operator that terror that cube form that was shared earlier by a peer that looks interesting or like a service operator.

The idea here is that if you can get more of this stuff all in to Kubernetes you're not bridging tool sets.

Yeah then it gets easier because you can have a helm chart that says, hey provision this Terraform code and take the outputs of that and stick it into this community secret.

And then, you know, blah, blah, blah, blah, blah.

Do all this other stuff in Cuban is.

I don't know that many people.

I don't know anyone doing this, though really in production right now typically multiple pipelines that have steps that this part doesn't Terraform this part does the OK robinet is and trying to make this turnkey for net new applications doesn't.

I'll just man up and Yeah, I don't know anybody disagree with what I said there.

I would love to be a rock.

OK So here is CloudFlare CloudFlare workers have gotten a lot of attention CloudFlare workers are a little bit like running lambdas on the edge like AWS lambdas.

I am not like a super expert at doing this stuff.

But I think that's a testament to workers.

But because you don't have to be very hardcore to get a benefit out of it.

So let me go about how we've benefited from using workers.

And I think these are some common problems that others might have too.

So like you know we launched our podcast at a podcast club posse and that's through buzz sprout and buzz sprout offers branded kind of micro sites for your podcast.

Lots of other services do this too.

For example, get review what we use for a newsletter that also offers branded sites.

Thing is how do you have all these different micro sites that they never give you the ability to customize the menu on them.

They never give you the ability to change the CSS on these things yet is your brand.

You want to have control over those things.

So how do you do that.

Well CloudFlare workers makes this pretty awesome because you go over here like to your DNS settings.

And when you set up like the DNS for look load here for a second.

All right.

So let's take a look at the DNS for our podcast.

So here is the podcast.

No, that's the record.

Yeah here's the podcast.

So you see it's a C name to the buzz sprout a year old.

But it's proxy.

So these all requests are going through CloudFlare.

So if I go over here to work here.

I can now set up a rule that says podcast star first goes through this work right here.

And let me just make sure unfortunately, there's no really good secrets management thing here.

So let me just make sure that I'm not going to leak any secrets by opening up this example.

Yeah muted or I hit mute somehow.

Thanks for pointing it out.

All right.

So what we have here is an example of where we're going.

We're rewriting the content on the fly that we get back from the upstream requests from CloudFlare.

So what I'm doing here.

The objective is that buzz sprout.

They offer this podcast stuff.

But they hijack the canonical link.

So that they say buzz Broadcom is canonical for my podcast yet.

I have a branded site that's not cool.

So what did I do.

I just configure the CloudFlare worker to rewrite that content on the fly.

So here I rewrite all a tags linked tags meta tags on the fly when I see these patterns I just replace it with my podcast.

So that it gave me full control over the content there.

Which is cool.

I can also do things like rewrite the rewrite the site map site map XML to 0.2 links in our own site as opposed to their site.

So that was some of the examples there that we did.

Another example.

That's really cool is.

Let's see here, if you need to dynamically install headers like content security policies.

You can do that on the fly.

For example, we're using the Slack in plugin.

So the slack in no JSF for Slack invites.

We could go modify that code.

But then that's just more overhead that we got to do when we just want to do simple things like maybe at the end or so years.

The ability that we can do that on the fly for those requests.

Very simple little script streaming optimizations.

No, I want to go to slack archive redirect.

So many of you run single page application space you deploy those to service like s three, you're limited in what you can do.

So if you wanted to do a redirect it's maybe a little bit harder to do so very kind of pattern is you have a stub file that has a metal refresh equip in there and that made a refresh does the redirect but that kind of sucks because you first see a blank page and then redirect and it's not good.

So here's an example where we fetch the upstream.

And then look for that tag.

And then we do a rewrite automatically at 301 and serve that back to that.

We see that rejects matches.

So that's how we split up our outside.

We actually see the console logs.

Can you see someone somewhere.

Or so.

Yeah So what's cool is the way they've done it.

I mean, if I go here to I got a change domain.

In this case sweet UPS and then go to.

So here's this route I've configured everything to go to slack.

Archive redirects I can launch the editor.

I'm going to open up a slack archive redirects I'm going to go to that domain here.

Mobile are still all the console logs and everything.

My debug output I see here.

OK So this makes it really good that you can just iterate very quickly, you can update the preview as you're updating your code here without deploying it and seeing it.

So it's a safe kind of developed mode.

The downside is they don't provide any access to logs period.

Cloudflare it's not a server.

So what you got to do is you gotta integrate this century or some other third party service where you can basically post your exceptions in your log events and stuff to some third party service be related to this.

There's another similar feature that's existed in CloudFlare for a long time.

But it was implemented using workers.

I believe a behind the scenes.

But not now.

I'm just truly in love with it for these reasons.

So if we go to apps.

They have a marketplace on CloudFlare with all these hacks.

It's like a marketplace of hacks like the different blades of your Swiss army knife and there's a few in here that are really special.

So if I go over to install apps one is you can inject a site.

Now onto any page, you can add CSX onto any page, you can install a Tag Manager on any image, you can add each email to anything age.

So you can see how you can really do anything you want at this point with any page any site that's part of your network.

So like if I go now to cast that cloud posse I have a navigation menu here that's not provided by them.

That's provided by CloudFlare that drops me over to the newsletter.

Would you look at that.

I got a navigation menu that's not provided by them as provided by CloudFlare.

So all these things.

Let me stitch together all these different properties provided by different services that micro sites and get review.

They had a really bad CSX problem that made the stuff look like doesn't look beautiful.

Now Don't get me wrong.

But it looks atrocious before.

And I was able to inject mounts yes to clean this stuff up.

That's called the vertical wow.

This is amazing.

Yeah So CloudFlare is pretty rad in that sense.

But there are other examples.

I'm not going to say names of services that do this, but there are certain sites that can go to a pay up if you want to have a branded hostname.

Not anymore.

If you use CloudFlare you can change the origin of your requests and you can serve it under any domain you want on your site.

So you can do your own branded versions or micro sites of services like that.

Oh, wow.

This is under the free offering too.

Yeah you basically.

And distinctly CloudFlare.

I mean, God bless them you know, they are really fair in their pricing it's not that they have also overage pricing.

So you can start for free.

And then they have a reasonable rate that you can pay per million requests or something like that.

I hate these services that require you suddenly to you go from spent $50 a month.

Then they want a one year commitment at $1,500 a month.

How does the basic fee and compare to front say that when we're done.

It's just a basic TDM capabilities.

How does that compare it to something like we crowdfund.

Well, so this cloud front is pretty easy to beat that.

That's probably the worst CDM functionality wise that I've used.

Now it's gotten a lot better with the edge lambdas on cloud front, but the fact that any change you make to cloud front takes like 30 minutes to propagate.

That's a massive business liability right.

Like oh, we made some mistake an honest mistake.

OK this is down for 30 minutes.

I don't like that was pretty rad with CloudFlare is changes take seconds or less.

Most of the time.

And they manage it to Terraform or manually.

I confess we personally are not for these things.

But CloudFlare is investing a lot into their Terraform providers.

And I believe a lot of this stuff can be done with their form.

They also have a command line tool I for get what it's called, but they have a command line tool to manage this stuff and be more command line driven perhaps somebody can correct me maybe.

But I don't believe CloudFlare has scoped API keys.

And that's kind of like scary.

So Stephen Dunn fought for companies that want to use Terraform with CloudFlare and control kind of the blast radius of changes is you can actually use proxies with Terraform right.

So then there's a company called b.s. very, very good security and very good security is pretty rad because they'll be your man in the middle proxy and that you tokenized is basically that request.

You can use dummy credentials.

And you can scope requests to certain parts basically certain API endpoints.

And it will inject the tokens only for those requests only from IP is that you want only using credentials that you set and those credentials.

If x still traded or leaked are useless to anyone else.

So Yeah Terraform with very good security and CloudFlare is kind of like the pattern there.

We're almost at the end of the hour here.

So I'm not going to do a demo of this, but it's interesting.

We can do something arrange it a better demo of this.

The idea here is like you can create a route for like API that GitHub you can whitelist a certain IP range.

And then when these conditions are met.

For example, the path is such and such, then you can do replacements on fields in that request.

And since most service most things support the HP proxy convention, then you can just have this happen transparent.

All right Erik do you know anything about the kind of phone data protection laws that go into effect beginning of the year.

The GDPR stuff.

Well, I don't know how you call it in California.

But I think it is the equivalent of the same.

Yeah, we haven't been doing that much.

We haven't yet.

We haven't you know buy our customers been asked to help them with any of that stuff too much.

The cool thing, though, is with CloudFlare you can implement that across the board automatically.

You can install those retarded little pop UPS.

You have to have now on sites that say we're exporting cookies as if you didn't know.

And those cookies store information.

Well, with this if they click No CloudFlare can automatically always drop it.

So you can't track them.

Yeah There's some issues.

A bank in Germany implemented Taillefer last weekend and didn't update the data processing agreements on the back page and didn't inform the customers because they're around the details attack and basically counter.

They did so as to ask termination thing.

And now all the customers are pretty pissed off because they gave closure over all their banking information.

But that's the thing that think that happen happened Germany, and it's a pretty big GDP violation when it did, they screwed up the cash rules.

So they were cashing private data or not.

But I think just we'll see initial things that happened.

But they didn't inform the customers that there is now a middleman.

You have to do.

You have to have data processing agreements with the companies and they have to be accessible for your end users.

Yes And they did not do that.

And yet.

Now that 30 is that is really interesting.

So I don't know.

Yeah somebody else is using CloudFlare and you can share more information about that next time.

Maybe it really is.

All right, it looks like we've reached the end of the hour.

And that about wraps things up.

Thanks again for sharing.

I've learned a lot this time and that some others got the problem solved.

So that's pretty cool caught a recording of this call will be posted in the office hours channel.

We'll also be syndicating it to our podcast.

So see you guys next week same time, same place.

Thanks a lot.

All right.

Public “Office Hours” (2020-01-08)

Erik OstermanOffice Hours

1 min read

Here's the recording from our DevOps “Office Hours” session on 2020-01-08.

We hold public “Office Hours” every Wednesday at 11:30am PST to answer questions on all things DevOps/Terraform/Kubernetes/CICD related.

These “lunch & learn” style sessions are totally free and really just an opportunity to talk shop, ask questions and get answers.

Register here:

Basically, these sessions are an opportunity to get a free weekly consultation with Cloud Posse where you can literally “ask me anything” (AMA). Since we're all engineers, this also helps us better understand the challenges our users have so we can better focus on solving the real problems you have and address the problems/gaps in our tools.

Machine Generated Transcript

All right, everyone.

Let's get the show started.

Welcome to.

Office hours.

It's January 8th 2020.

Can you believe that.

I just can't get over that it's 2020.

My name is Eric Osterman and I'll be leading a conversation.

I'm the CEO and founder of cloud policy where DevOps accelerator we help startups own their infrastructure in record time by building it for you and then showing you the ropes.

For those of you new to the call the format of this call is very informal.

My goal is to get your questions answered.

Feel free to amuse yourself at any time if you want to jump in and participate.

We host these calls every week will automatically post a video of this recording to the office hours channel as well as follow up with an email.

So you can share it with your team.

If you want to share something in private just ask when we can temporarily suspend the recording.

With that said, let's kick this off.

Here are some talking points.

I came up with.

We don't have to get to all of them.

It's just things I want to bring up and first thing is that we now have a syndicated podcast of this Office Hours, which is really cool.

It's so easy to do these days with things like zap gear, which relates to one of the other talking points.

The next thing is we also have the suite ops job board.

There are a lot of companies hiring that are in our suite ops community.

Our goal is to bring everyone together and that this falls in line with that.

So if your company is hiring for DevOps or something adjacent to that.

Let me know and we can post that to the suite ops job site.

Totally free then searchable slack archives.

So if you are in our slack team, you'll know that we are a free team.

So we're limited to the 10,000 messages but we do an export of that data and we posted to the archive, which is archive that sweet UPS and we've just invested in adding Angola to our search index to that.

So it's a lot easier to discover the content.

And we'll continue to invest in that and some other things.

The other thing is that public service announcement go into this in a little bit more detail later, but basically, AWS has announced that the s.a. the rootsy a search for RDA straw raw and documents will expire on the 5th of March.

I expect like a little mini Y2K kind of bug therefore, companies doing cert validation an arduous in March and then Andrew as Andrew Roth on the call.

He is not well Andrew had asked.

So we might push this.

He doesn't show up.

He was curious about how we are doing some of our slack automation using zakir as a pure enemy, which it is.

So I can show that.

And if we have time.

How we use CloudFlare is kind of like the Swiss army knife to fuse sites together inject content and do all kinds of magic with workers.

All right.

So with that said, let's get the open the floor up to anyone.

Anyone have any interesting comments or problems that they're working with.

Hey, everyone.

This is Brandon.

Long time no see it's been a few weeks since I've joined.

Just a quick question for his for people.

Does anyone have an that they can recommend.

I'm looking at our annual PCI external scans coming up and looking for a recommended RSV.

Anybody has one or some people.

And the PCI community, I can probably ask some friends if they have recommendation.

Yeah, just kind of looking for somebody you know trusted and decently well-known.

Yeah which area are you here in Los Angeles.

Well, I'd be happy to make an intro to any ocean.

They've helped out some of our customers and have been very pleased with their services.

It, man.

Brian Neisseria.

I'm seeing you join for a while.

How's it going.

Yeah, I always come to the country in December, I spent time with family.

Did you.

Where were you able to get to the bottom of your issues with Kubernetes and the random pod restarts.

I think the actual cops work or know to be starting oh, actually yes items cops work for nuts restarting correct.

Yeah still I can't quite pin it yet.

Nothing in the logs.

Nothing in the kernel logs.

It's weird like you'll just see like the Cuba logs in the kernel logs like they'll just go.

And then they'll have like one minute of one minute of nothing in the logs and then they'll just say reboot and then have like the new boot on the walker no nothing.

And easy to the system status checks everything good.

It's interesting just because like, we cannot find any logs that show why the reboot happens, we just can't see when it happens.

Did did you go about implementing those resource limits on the pods.

I did.

Yeah So now everything on those boxes have resource limits.

Limit on requests.

OK And Yeah.

Especially on memory.

And now are you tracking how much memory is being consumed on those boxes and can you see it reaching the upper limit.

So it does reach the upper limit.

Sometimes per pod.

Well, I haven't quite figured it out or I haven't quite tuned perfectly.

It is.

We have the historical historical data on not how high a part can go in memory, but now the question is, at what point do we say this pi is API We should kill it vs. this pi does need to use this in our memory for this time being, we should leave it up, because I don't want to add the limit be too low to the point where our clients are running into issues in the app because the party's getting killed right.

You don't want to artificially reduce the amount of memory to the pods to satellite.

On average, it will consume 1 gig of memory sometimes.

But historically, the data shows that sometimes these pods can go up to 5 gigs in memory.

Do I cut it off at 5.

Or is them going to 5 mean that something has gone wrong on that part.

That's a hard one to answer.

Does anyone have any insights on this.

Have you said your workload is it.

Well, welcome to a what kind of workload is it.

Yeah, this is just simple IV application.

You get on with like what kind of data those and all being is it it spike because users are uploading 5 gigs of data like some.

Yeah, we seem to have like the dependent.

So the arbitration dies does stories like Excel files and some of the files aren't started S3, but like a story like a modified version of the Excel file in the database.

So there are probably large amounts of data being sent up periodically, but not consistently.

So that will cause the spike in numbers.

I mean keep if you haven't already found a correlation in the metrics, then keep tuning the metrics until you find one.

Here's what I would say that it's a good strategy.

So going back to what you were saying though you don't know where to set that limit.

I don't know.

I don't know what the impact to the business is by following the advice.

I'm about to give.

That's what you're going to have to decide.

But what I would say is set the let's set it at what seems reasonable on average and not for the peaks.

Allow some pods to crash OR gate kill the reaper every now and then for the purpose of seeing if this address is the stability of the cluster itself.

If you're problems then go away that you aren't killing the servers.

Well, then at least you identify the problem.

I think it's better to not cause a reboot of the entire server than to lose a pod every now and then to know memory bursts.

Yeah, exactly.

Did you set up.

Do you have the Kubernetes autoscaler setup.

No, I don't.

But at the same time, these are these nodes are already massively under scaled in my opinion.

Oh, yeah CPU usage is at 40% And memory usage is at about 50, 0.

So it's like these nodes do have enough resources because my only concern is that when you set the memory limits that.

Now, you might be constraining pods from being scheduled.

I guess the limits are if the limits are too high, that will constrain them from being scheduled using.

Yeah if the limits are too high and you don't have free resources because it's actually the op the requested limit.

It's actually allocating so that that is reserved and can't be used by the parties.

That's the request.

Yes the requirements are not the limit.

Oh, yeah complaining that too.

Unless you have a limit ranger that has a max limit, then the limit shouldn't I have two requests that lower, much lower or and then the limits higher much like the request I have to attend to the average memory usage of a pod.

And then the limits.

I have it to the peak.

OK on the topic of pods I have a question that I've asked in suite ops.

I'm curious if anybody has run into this.

Just I have no 1800 pods on those of a specific type running on 60 nodes.

How do you get those pods to be distributed close to even the on each node.

I feel I've seen something.

Basically you're looking for something that does rebalancing and I'd have to look that up.

I don't remember off my head my issue being like, I have a pretty large difference in memory requests on some node versus others.

Some notes are up to the 90s, and then some notes have 40, 50.

So it's like I want to even out.

I don't think I need to scale up more nodes I just need those nodes with 90% 95% memory to give those type of pods to denote with 40.

I mean Cuba is humanity's tries to do that automatically, but for new parts right.

Not for existing pods is it.

I don't know.

Also I mean, if it falls, it won't kill a pod unless it's told to unlock you know if a pod is Mayor happily running.

And you know unless you have some other functionality that is going through and saying, I want you to go and go around and start rescheduling pods.

By default Cuban edit once you schedule a pod.

And it's there, and it's running on a node.

It's going to keep running.

Yeah if that pod dies and gets restarted there's no guarantee it's going to restart on the same node that it was before.

So one thing one thing that I'm really interested in that, I haven't tried yet myself, but I'm I'm incredibly interested in doing for two reasons.

Number one is what we're talking about with the efficient bin packing of pods across the nodes.

But the other reason being the enforcement of your immutable state is automatically restarting all pods in the cluster, or you know whatever pods you have in a white list every x number of hours.

I was talking to I was in the Ask Me Anything section that Nick's salon did for the DVD and they're doing it in their deficit gaps.

You know, high security clusters that they everybody gets restarted average every four hours.

That's very interesting.

I found not fire right now because I'm too lazy to write a prompter across the grease scopes helm shops on startup.

So I just kill it every couple of minutes, hours.

Well, we're using for that.

It's just time off.

OK Oh, yeah, you're at.

That's a smart trick right.

So what you see is your entry point script if you just add that it just add the timeout command.

Then you can have your pod automatically cycle itself.

This is optimistically right that, of course, they won't all do it at the same time, which is unlikely but good luck.

And also, it's a hard kill of the process.

I imagine.

So I give you all the existing connections.

So we'll just go, Yeah Yeah, it's not great since long timeout is actually six term.

Let's see.

Oh, OK.

So you can actually kind of let in the and past.

Yeah you have handles the sig turn gracefully.

Let's go.

Also I'd say I shared to office hours.

It looks like there's a project out there.

It's called scheduler and it will wake up when it thinks the clusters unbalanced and start killing pods off.

So that they get scheduled I would argue that that is not super necessary.

Because if I mean, yeah, sure if a node is getting full but everything's running fine.

So what Yeah.

More or less.

So right.

So if you have an overloaded node overloaded being a subjective term.

But let's say it's running at 80% capacity everything's hunky Dory, and then Brian's cluster.

This one pod gets that request that causes the memory to spike 5 gigs.

That's what he wants to avoid.

So if on average, it's more balanced.

I think the probability of that causing problems is less and we're assuming that that misbehaving pod doesn't have a limit on it.

Well well now they do.

Now it could still be all year.

And 90 like 99 or 96 percent.

Memory requires you don't have much of a memory left at least for me, I have one or two accidents.

Yeah, I mean so not like it's not like all my notes are the 99 96, 97 percent.

But some of them are.

But then there are on the other side, there are other nodes that are at 30% 40% request.

So I want to be able.

Like I want to get to a point where obviously all of them are like around 80% memory requires a 20% pool of memory for when you do spike up to five peaks in memory.

I do see these spikes but very short.

So that's two I 80% is too hard.

Yeah, I think 8% to my clusters are like 40% memory request, maybe that memory.

40% memory usage memory recall.

I don't know.

I'm not sure what's a request sir.

So for me, the I have nodes between 30% request memory request and 99 percent memory request.

I'm struggling to get them all to like 70 80% memory replace actual memory usage on each node is about 40% 50% actually.

No, I take that back 50 60% I do think this is good.

But I would say you know to make a general statement, I would say, don't sweat it too much.

You know unless things are going well.

Oh so.

Because I have these nodes that just reboot every week and a half randomly.

I have AI.

This is my best theory right now is that every single time.

There's no limits their memory requests are really high.

It's in the 90s every single time.

OK So I mean, because I can't find anything in the kernel logs that Cuba logs.

This is my best bet right now.

That is it's kind of why I've gone down this rabbit hole.

I'm not 100% sure if this is the reason why it's happening.

But it's my best bet right now.

Just because I don't see anything in the logs the other interesting thing just a theory.

But I'm not sure if it would help would be if you did have the autoscaler enabled.

What you will see is nodes more frequently getting replaced.

So that would automatically also impact the rebalancing of pods across the cluster.

Now that may or may not alleviate the problem.

For example, if you're using spot, like the service you would be replacing part nodes all day long every day, throughout the week.

And that would also keep the OS fresh on all those boxes and the pots continually getting rescheduled I have these ephemeral clusters.

So like some of these easy to notes or four or five days old, and then they'll reboot.

So it is.

Previously, we had clusters that were up for longer, maybe three, four months and we'll get that maintenance reboot, which is on the hardware.

But you'll see that in my cloud watch.

You'll say that you see two system health, whatever or whatever our system size check cause a reboot.

This is like a service level reboot.

Not easy to host reboot.

Any other questions related to Terraform DevOp security.

AWS I've been doing.

My good friend.

I'm going to start digging into the AWS SEC s web example a little bit on preparing a eight hour training session for there.

And perhaps the next.

Once I dig into it a little bit more.

I have questions, but I didn't see an ek as examples.

That's why I just swing straight to VCs one.

Oh, we definitely have examples of all of that stuff.

Let me show you that a full screen using this goes for pretty much all of our modules especially those that have been upgraded to Terraform zeroed out.

Well support.

Yeah, that's kind of why I'm not using my packages because I don't really want to bother with China.

And yeah.

What better way than to also start contributing to.

So like here.

Let's take an example, at the UK case cluster.

So one of things we've done is under test and source we've implemented some basic terror test examples that bring up the cluster ensure that the nodes connect.

And then tear it down when everything is done or if it fails.

So here you can see like the logic for waiting until the worker joins the cluster.

This has enabled us then to accept all requests much faster for the modules.

Now, if you go into the examples complete folder here is where we have.

This is not saying this is a reference architecture is a minimally viable example of how to get the UK s cluster up.

That should be sufficient for your class.

So as you know, our modules are very composed tables.

So then when you first bring up the BPC with provisioning subnets then we appreciate the workers and the UK cluster and then the this worker node pool is then connected to this cluster of a suite.

I'll take a look into this.

It's I'm not sure I want to overwhelm my students with all of that at the same time.

But I definitely want to reference if you are interested in using Cooper that is the kind of go down.

Yeah communities.

A lot of complexity there.

And I mean now as it relates to yes, I think they are easy yes implementation is more on the advanced side of how to do.

Yes and isn't it like the Hello World example of this.

Yes So just keep that in mind as well.

I know I'm kind of unsettling our module there.

But the point is that our modules are designed for advanced use cases and have a lot of degrees of variation configuration.

Yeah, I agree.

It can be tough.

I learned yesterday a module.

But I end up writing my own after that, especially like with the test definition that you have.

It's kind of funky.

I hope you do you guys do it, you do a module.

So it might be better to just let them know that this is a complaint.

I have about using ECM s in general, especially with CI/CD.

The fact that you need to use a heavy handed tool like Terraform to deploy a new task.

It's like the wrong tool for the job.

Yeah, I had quite an experience.

Yeah, I work with CEOs deploy, which is a Python script for the front, and it just like don't deactivate your old tasks.

So Terraform farm sells things that they are active in and won't try to modify them.

So yeah, the trick I did for OCD is not ideal.

But it works.

And this is why I fight Yeah.

So this is one of the complaints I have with these.

Yes And Terraform it is that deployment part in this is this would be true if you were also choosing to use Terraform to do your deployments to Cuba natives, but many of us do not use Terraform for those deployments and therefore, we don't have that same challenge.

And the thing is, it could be quite easily solved by Terraform.

The only issue they do is basically, if there is an inactive instead of taking the latest task definition, they just stick to the one they have.

And if it's inactive then consider it dead instead of like looking up inside all of that as they finish an on line and then just taking the latest one.

So I think that's a sharper strategy strategies that it could take that would be nice.

Yeah, I appreciate yells help on that.

And definitely will be checking back in once and start digging into a mine.

Yeah, feel free to ask away in the Terraform general, if you blocked.

I'm sure our Savior Andre will help out.

So then the vacation time I got into an argument with some friends that needed some help.

This community.

This cluster and they wanted to scale their services.

Some tons of close stuff and based on latency and I was like straight extending a latency something that I've never done.

And would never do.

And then we I. Why is this a bad thing.

And I just wanted to know how do you scale.

Like imagine provisioning and HP.

How How do you scale on which kind of metrics.

I guess superior memory is the most common one.

But I think latency if you have it don't seem at to the database.

Exactly I mean, that's why I was like, no, not good.

But they were pretty light persistent on it.

I was like, that's something for an alert but not for getting your pompous Yeah, I just would like to you input because I mean, I haven't.

Well, I guess they are going to open and we decided to play something out.

But I mean, it's kind of like, oh, did you guys like that.

I would just like to know.

First of all, the arguments you have during the holidays are very different than the arguments I have during holidays no politics none of them different kinds of family discussions.

Now you brought up the great point right.

Like in an academic setting like devoid of reality and all the other things that can go wrong scaling a latency seems like a great idea, except for in reality, it's usually because of all these upstream problems that you have no control over.

And like it's the data we a down your latency goes up.

So you scale out more workers, which slammed the database even more as it slows down even more.

And then you have a catastrophic failure.

Everything falls down.

So I think that when you scale or how you scale it.

This is a cop out.

I hate to say it.

It depends a lot on what you're building UK sees a TensorFlow right.

Let's see.

I thought that kicks of memory allocated to them.

I think it was for use.

So it must pretty hefty parts.

Something's wrong.

So let's see here if I remember this correctly.

I was a year and a half ago or so.

We actually we did a project with Caltech and using cops and EPA and TensorFlow and a have a reddish q with all the tasks in there leave or the images or something that they needed to process and they then rotate custom HIPAA as autoscaler there's a grid.

There are a lot of examples on how to do a custom setup for that.

And so they wrote a custom one that just looks at the size of the Redis q and then scales the pods based on that.

Yeah OK.

This one's more like a traditional application thing basically sending data.

Why a post.

And then it was doing language processing while also saves on a lot of lost in processing then a lot of traffic they had like 50,000 requests per second.

I don't know what it is.

Yeah, I don't know what aligning.

I didn't ask.

But it's like, yeah, the spots where image processing that's what the language processing and the eta plus I had 140 notes or something.

All 64 kicks a friend was kind of crazy.

But yeah.

So yeah, just like that.

Yeah, it is.

I was like, I was really interested in it.

But it's like, yeah, my counterpart was a scientist.

And he was like area using two to are aside each and doesn't have time.

I want to have.

I was like, OK, give me a contract.

I will sign and give you money.

And I will look at it.

Wrote quite well.

But it's like, yeah.

Is the argument I had.

But it was kind of like that.

Yeah Yeah.

I think at the scale that they're doing this example kind of isn't going to be helpful.

But for what it's worth here is that auto scaler and that stuff is all defined in this kiosk.

I'll share it in general office hours.

Jim OK.

Any other interesting questions, guys.

I have a question like him regarding communities like if you have servers that require direct mapping.

I have an ambassador in front of it.

So I was wondering like, oh easy or hard it is to get the raw socket there for a couple thuds and then doodle bunting.

You should read these suckers.

Let's see if I just to say to my own words you want to, you want a dedicated IP first service.

I don't really care about it.

Nothing I didn't understand lol.

Basically I have like a couple servers that are only raw socket connections.

Yeah I'm trying to root them using ambassador while location really matter.

But like something in front of it.

So they can act like a unique IP with a different public port and interest not due to the internal port gotcha.

I don't have any feedback on that one.

Somebody else want to chime in.

Maybe with the ambassador experience or alternative energy mix.

Or unbought or what.

Sorry Yeah.

Hey Eric Garner's going to question anyway.

Do you play with kids from Windows.

Does work in those games.

Mp one that I'm about to get I'm working on that.

You're a brave man coming here like that.

I'm kidding.

I know I saw you created the channel windows.

It is sweet ops.

Thank you for that.

I guess it's about time.

This is not the Yeah, it's not the same OS used to be.

But I can't help with that someone else you're beautiful.

Just about anybody stuck in it.

But what I'm asking.

Yeah, I'm dreading the day I will have to do that.

It's coming though.

It's in the future.

Yeah well I'm driving.

So what is it is what it is.

The last time I had to do windows knows I used as Azure Service Fabric and it sucked.

Well, let's see here.

So Andrews as you're on the call and you'd asked a week or so ago or two weeks ago over the holidays kind of like how do we do these chat notifications actually.

Perfect segue way because Carlos was the one who recently created this channel here.

So how do we set up this automatic notification when new channels come up.

So it's a pretty quick and easy demo.

Basically here I have a link here somewhere.

So do many of you guys use active today, or savior.

I never say I use it for our CNN integration with one of our products.

My mouse has.

So yeah.

This is just a deep link into our accounts.

I don't have to search around for it.

This is the app that we have that handles that.

So basically, with the zip here you can have all these different integrations to the products you use Gmail slack MySQL plus grass GitHub et cetera, et cetera.

And in this case, the application is when a new channel is discovered via web hook integration, then it will.

Here's an example.

When I set this up a year ago.

But then here's the metadata that came back and who created that channel.

So then we can just jump over to the next message, which with next step, which is to send a custom Channel message.

And what I love about it is you can really customize everything about that message and how it gets sent.

And it makes it very easy to just add fields that are available from any of that data that was passed from previous step.

And we've taken this to the extreme, you know in our account we have here in our account we handle almost 20,000 zaps every month.

And we have over 111 of these across our company.

And what we do to automate various aspects from things that happen in Gmail things that happen in QuickBooks things that happen.

And build out com all that we automate send it into our Slack channel as simple as view for what's going on with Social Security's app and a task could just be overloading.

Oh, OK.

Yeah, exactly.

Exactly I looking at the pricing and it's like, OK, the free play and you get five apps and 100 task execution.

So a zap is the is like the code is the program.

OK you get unlimited programs.

And then how many executions of that.

OK So it is another way of thinking about it, especially now that they have code steps.

So you can run JavaScript.

You can run Python code.

So it's almost like just a goofy for doing lambda and then they provide all the integrations to all these different services.

And it becomes the glue to tie everything together.

Yeah the downside since this is a ops oriented group.

There's no API.

Ironically there's no API to zap here.

It's really strange and because there's no API.

Exactly there's also no zaps to automate zappia in that way.

Which is unfortunate.

So you can't do zap erase code no Terraform module or anything.

Well, so disappointed.

What happens when you run out of your 20,000 packs or whatever.

It's a shelf.

It sucks.

They require you to upgrade to the next plan.

They don't have an overage model.

So if you.

So if by the 16th.

I max out this 1,000 zaps here.

I'm going to have to upgrade again.

Another $100 a month or whatever.

Yeah Yourself though.

So we create 1000 channels small and well I say reset.

So we create one, I want to go create an idea.

Don't do that.

Look look look.

How can we cost Eric money.

Let's do that.

OK Yeah.

That's funny.

OK How much.

It seems like that.

It seems like it seems like it would be not that hard to write to create, a lambda or something that you hooked together with a broken leg.

A true developer my friend.

Spoken like a fruit to developer.

Could you really just land.

You can with slack.

You can set up web books.

Yeah, no.

And the reality is there's a open source.

Zap your alternatives.

It's feel you know buy versus build.

Mentality it was just posted on Hacker News.

There's in good students or Huggins which h right here can get something.

Well, I see it there.

Hug Im I think it's huge.

Thank you.

The agents that monitor and act on your behalf.

Like I'm so blind here.

Agents that.

Yeah Georgie I and I just skip it.

Just go back to just this is the funny thing.

Yeah, just go back up there.

You got a different hash.

Wow Yeah.

Linus wow.

It wasn't taken back two months ago.

A woman.

Yeah So yeah, I think it's one of those things depending on what you want to accomplish that could be great.

Also a lot of the zap your integrations are actually on their GitHub.

I just realized the other day.

So they're open source as you can PR them if you need to be interesting if something like human actually worked with zap your plugins but I don't think it does.

Yeah, it's also not pretty.

Oh, it's been great.

I was like plugged in any moment that you added.

Yeah So let's hear it.

Here are the agents that it supports.

Where is the screenshot.

So I don't know how many agents.

It supports today.

I think that's the main thing.

The main consideration is like when you go to apps here literally, I think there are thousands of integrations that it has.

1,500 So it's quite a lot.

And anyone who's worked with third party guys knows they break all the time.

They upgrade they change.

So you might get it working on a weekend.

But it's going to break three weeks from now.

And then you're never going to get back to it, because now you're not on holiday break right.

OK, cool.

Thanks Or any other questions about this or something else.

Nobody using pot man yet.

No, I know they came up a few weeks or a month or so ago.

I know officers this is always a serious.

If they once kick the tires on this project.

I think it's in the incubator, a cube sphere.

If any of you guys think it's here.

So when I got this dashboard.

I want.

Yeah Yeah.

But it combines a lot of things.

And it's kind of interesting.

If you look under the hood on the architecture of what it uses.

It's pretty similar to the components that also first of all.

Here's the UI.

So you know pictures are always pretty.

So it's a nice UI of everything operating in the system.

You could say it's maybe somewhere.

It's something like an shift.

But maybe not as opinionated or as complicated.

I believe it deploys entirely on Kubernetes.

I like shift, which you have to have control over the most US, they have a demo environment you can log in and check out.

But here's the architecture.

I thought was interesting.

Let's see.

Why is not living it looks like ranger or ranger.

Yeah like, yeah, I'm looking at the I'm looking at the screenshots going to wait.

This isn't a rancher.

Yeah, looks like they're similar.

It looks like the GitHub Kamau proxy has bungled this image here is like maybe you can I. When you open up in the incognito.

So glad you're in production yet ranch though uses mostly their own in-house components, though, right.

And isn't built on.

Aside from communities, of course, which are built on, but it's a lot of in-house things versus this.

It looks like.

And I can't see what you would see here is that it uses like Jenkins under the hood, it uses Prometheus and the fun.

And I think it's had fluidity and it looked like very much the similar stack that we were using today.

I was going to deploy the not operator and I saw it.

Also, because it's on computers and then it's kind of knowing that everyone and everything is deploying its own dependencies.

A little bit.

I kind of like I it all the time.

And knows me.

I don't know where you're deploying.

What was it nuts.

It pops out alternative and operate off nets also deploys parameters and co-founder and all that stuff.

And I like my reach as operator.

What was it like.

There was one a bundle Prometheus with it.

Yeah Fox is doing the same.

It's like, yeah.

OK, I get it.

You need it, but it's like it's always included right now.

It's like so.

So from experience.

I think I have the answer.

Now to that.

But I am equally frustrated.

So we support a product called cute cost.

We have a channel 4 that's in this meetups that keep cost channel there and cost is a cost.

Like this visualization tool for Kubernetes, and your cloud at your cloud provider.

So like all sports to get and we'll show you how much that's costing in real time estimations of course.

But if two ships its own version of Prometheus and I shook my fist at them and said, why are you doing this to us you know we spent a lot like scaling Prometheus is non-trivial.

You can't just throw Prometheus and expect it to actually work in production.

You can do it for a great example and make your charts very presentable but it's not the way forward.

And then worse yet they didn't support a way to have your bring your own Prometheus so that they made it worse.

Well, we spent probably like a month on the integration effort trying to get there.

Keep costs working with our Prometheus and it just there's so many settings and there's so many assumptions and there's so many things that make everyone's Prometheus installation kind of different.

Especially, I guess if you're not maybe using Prometheus operate.

So I guess the reason why all these vendors are shipping it is because most people a lack for me is operational experience.

We know how to set it up the right way and see they got to have a successful product install in the first five minutes or people are just going to get bored and walk away.

So I guess that's why Nasa is doing that too.

And I can only assume that it's just like, that's fucked up.

I mean, it's not even using that scanning or something just like monitoring it doesn't have any use.

It's not utilized at all.

Only scaling something into just like it's there.

So I guess I guess I guess what we got to get more comfortable or not the right word.

Maybe or so.

I mean, the built in Prometheus the architecture is pretty nice right.

Prometheus can scrape other Prometheus is.

So I guess what.

We have.

So I guess really what we should be adopting is a pattern of more getting more comfortable with just scraping other Prometheus is an aggregating it rather than expecting all these third party services to use our Prometheus Yeah.

Just looking at the memory using obscene.

Can we use it open positive post, for example.

It's not much.

It's a 150 megabyte and being about.

But it's like still it's just an awesome idea Federated and died scraping basically from my Prometheus operator and it still is like one more component that you have to worry about.

And it's the same component.

And if someone tells you.

Hey, my computer since I'm working.

Which one is it.

Yeah Yeah, right.

So if you guys use titan or other like aggregator for Prometheus.

Like storage baggins.

Yeah Yeah.

Yeah, it's more like a federation also.

No, I only belch up.

I Volta as I function to actually save everything to pop search explode in search of search to have it acceptable that this came up recently also when talking with a customer and also I know Andrew Roth.

He does what we are doing and that I was actually going to chime in and give an update.

So now it's been probably two, three months operating.

Yes on the efforts for production cluster.

And it's working well.

But there were some growing pains along the way.

The two biggest problems we had was one was if you don't have the memory limit set correctly on previous overtime it will just crash and when it crashes, it means these wall files around and you can't Auto Recover.

After that, you have to exact into the container and delete as well.

But the other interesting thing we noticed from the Prometheus is.

So I forget what version it is.

But the latest version of Prometheus under Prometheus operator.

This is second hand.

One of my teams team members was relating this to me.

So forgive me if there's some loss of precision here.

Is that it will auto to discover what your memory limits should be for Prometheus when you deploy it, which is cool.

However, we noticed that when it was auto discovering those limits.

If we didn't have at least a minimum of 11 gigs available to Prometheus then it didn't matter what it set those limits to automatically it would always inch up against that hard limit and then get killed by the memory reaper on the Linux kernel.

So once we bump that up to 15 gigs it's now sitting under that limit and being respectful of everything and works.

We didn't dig deeper into why it didn't work when it was just under allocated.

All right.

Well, we have five more minutes here.

Any final thoughts or interesting news articles that you have on your agenda.

Well, we haven't talked about it yet.

There one thing.

But I don't want to I don't want to waste five minutes and rush through it.

I think you guys will dig this this using how to use CloudFlare as a Swiss army knife and kind of like we as the zappia for the same thing.

You can do a lot of nice dirty, nasty hacks with CloudFlare and workers.

And we're doing that now.

So I wanted to show how you can do content injection.

Page rewrites and proxy basically any site you want.

I did a little bit with called fertile not too long ago.

And I was thrilled with the quality of the Terraform provider for CloudFlare.

I was able to set everything up in Terraform yet.

Now I know they're investing a lot in that.

I think a couple of the people involved in that project are actually in suite ops and I know they've actually reached out to cloud posse on their Terraform provider for feedback oh cool.

Well, yeah, we are they we're on the radar one small project have seen this a couple months ago, I just posted the link in the channel.

It's a native policy management for communities.

It seems nice.

It's a company you like.

I don't remember which company, but they didn't give a presentation is seem interesting for, like you can set this policy, for example, every node needs to have limits set in your cluster or they have more complex examples.

And then it evaluates like which boards are in noncompliance with your policies and also blocks new deployments new policies.

So I thought it was cool.

Yeah, pretty simple way to express it.

Exactly Yes.

Well seals are native to communities.

Yeah star someone already tried out.

OK whatever complaints of policy agent because I wanted to do policy agent that basically trails off.

Basically, you need an approval that's only lead happening.

It's just like something you edit the just older proofs and runs through the eye stuff like that.

I wanted to try out.

Let's go down.

Yeah So some plans.

Now this is probably a topic for another office hours, but one of the things I really want to try and do with this Office Hours now in 2020 is more demos and more demos by our members.

So I know a lot of you are doing some cool things cool things that we don't have a chance to do ourselves or see.

I would like to see that.

So if you guys are working, for example, with OK or somebody kicks the tires on you know I would really like to get a demo of that.

Also maybe we can get some nice other speakers onto our person can ask them questions.

I really want to get you on here and wish you as prolific GitHub contributor to the community's ecosystem.

I thought about something cool that my team is starting to do a lot of work with some file and something that I don't think file has yet period that we miss from Terraform is that Terraform Doc's command.

Yes if there was like a hellfire Doc's command where you know where it would generate a table of all the environment variable inputs Yeah, that would be cool.

The cool thought about doing something like that.

Yeah, well, you know, at least the float the idea in the old file channel.

And let's see if there's some more interest in something like this auto documentation for help file.

Yeah, I'm having to write out the table right now because we were doing a big one for four GitLab and they're sticking point.

Yeah And we've got like 60 input variables for everything it can get unwieldy.

I agree with any.

And then the more anything, the more you pramod tries anything the harder it is to actually deploy.

We discovered that the environment's don't really work the way we expected them to and we weren't able to really use them the way we wanted them to.

If you're going to have a helm file and its purpose is to be a sub helm file meaning like I've got a helm file forget lab and the end user is going to create their own helm file and then it's going to have helm files pass you know get Cole and whatever GitLab get Cole and whatever key get go and would you know that if the sub helm file has environments in it your route helm file needs to have those environments declared and it can't have any other environments.

Yeah You have to do that.

You have to.

Exactly it's kind of an artifact of how file is implemented and the path of least resistance.

But yeah, those environments need to be declared if they exist in all hell and files where they get included.

Yeah, so we ended up scrapping that idea and just doing it because what we wanted to.

It was you know we talked earlier about how I don't I don't like how helm charts default to you know insecure by default.

I want secure by default.

So our home file is going to be the opinionated secure way that we want you to deploy helm file by default.

And then there is an environment variable called local mode or whatever that you can turn on that sets all the other variables period for you into the insecure mode.

So instead of having to settle kinds of stuff know by reading documentation and going have to do here.

Yeah run on my fucking laptop just do like local mode equals true.

And then running your laptop.

Yep So we were originally going to do like the default environment was production.

And then, of environment called local but that didn't end up working out.

I be happy.

If you want to share a little bit of that next call or in a future one I'd be happy to look over them and see if there's another way of doing it.

Or some feedback based on our experience.

Sure All right, everyone looks like we've reached the end of the hour.

That about wraps things up.

Thanks again for sharing.

I always learn so much from you guys on these calls.

A recording of this call will be posted in the office hours channel.

See you guys next week.

Same place same time.

Have a nice one.

But if you.

SweetOps Newsletter – Issue #3


6 min read

The SweetOps Newsletter is hand curated by Cloud Posse. We're a DevOps Accelerator for Startups. Own your infrastructure in a fraction of the time it takes to build it in-house. We built it. You drive it.

In this issue, we explore some of the new things we've launched over the holiday break.

Community News

Job Board

We've launched our jobs site so we can keep track of new opportunities posted by companies in our community. Have a job you want to post? Ping @erik on slack.

Check it out here: 👉

“Office Hours” Pod Cast

We've launched a podcast, which is a syndicated version of our weekly “Office Hours”. Tune in on the train or in the car!

Subscribe here: 👉

Slack Archive Search

You can now search

Slack Archive Search

We've added search functionality to our slack archives with Algolia. Now you can easily find solutions to common bugs or problems encountered by our community.

Kubernetes News

A lot of interesting links were shared related to the Kubernetes ecosystem. Here are some that stood out!

A Practical Guide to Setting Kubernetes Requests and Limits

Setting Kubernetes requests and limits effectively has a major impact on application performance, the stability of applications and the cluster, as well as cost. And yet working with many teams over the past year has shown us that determining the right values for these parameters is hard or worse yet teams don't even set them! Kubecost created this short guide to help teams more accurately set Kubernetes requests and limits for their applications.

Excellent Resource to Learn Kubernetes
Excellent Resource to Learn

If you want to learn more about Kubernetes, this is a phenomenal resource for learning the basics. For example, here's how to increase kubectl productivity when looking up resources, customizing the columns in the output format, switching between clusters and namespaces with ease, using auto-generated aliases, and extending kubectl with plugins!

CNCF Cloud Native Interactive Landscape
CNCF Cloud Native Interactive

There's an overwhelming number of cloud native technologies and open source tools. How do they all fit in? Here's a site that let's you filter and sort by GitHub stars, funding, commits, contributors, hq location, and tweets.

Terraform News

We've been hard at work upgrading all of our Terraform Modules to support Terraform 0.12 (HCL2). As part of this, we've added automated tests using a combination of bats and terratest. We publish all of our hundreds of Terraform modules in the official HashiCorp registry.


Terraform module to provision an EKS Fargate Profile to spin up a node group that run pods entirely on Fargate


Terraform module to provision a fully managed AWS EKS Node Group using terraform. Plus it works with all of our other EKS node pool modules so it's possible to run multiple different types of node pools in the same EKS cluster.


We've updated our Terraform modules for provisioning an EKS cluster to support Terraform 0.12. As part of this, we've added terratest to run automated infrastructure tests. Check it out here!

Want more news? Check out our Slack archives to learn what our community is all about.


SpotOn Senior DevOps Engineer
SpotOn Senior DevOps

SpotOn is a cutting-edge software company dedicated to redefining the merchant services industry. SpotOn, is hiring in either Krakow, Poland or Mexico City, MX for a seasoned DevOps engineer. BTW, they use TONS of Cloud Posse Terraform modules and Codefresh! =)

SettleMint Experienced SRE
SettleMint Experienced

SettleMint makes blockchain technology deployment, development and integration practical for the enterprise. Settlement is looking for someone in Belgium for a full on helm, kubernetes, helmfile gig (fulltime) in the Blockchain space.

What is a DevOps Accelerator?

Curious about what well built infrastructure looks like? Check out what we do at Cloud Posse.

Find out if we can help you by taking our quiz.

Free Weekly “Office Hours” with Cloud Posse

You are invited to our weekly Zoom meetings every Wednesday at 11:30 am PST (GMT-8). Join us to talk shop! This is an informal gathering where you get to ask questions and watch demos.

Register here: 👉

After registering, you will receive a confirmation email containing information about joining the meeting.

Public “Office Hours” (2019-12-18)

Erik OstermanOffice Hours

1 min read

Here's the recording from our DevOps “Office Hours” session on 2019-12-18.

We hold public “Office Hours” every Wednesday at 11:30am PST to answer questions on all things DevOps/Terraform/Kubernetes/CICD related.

These “lunch & learn” style sessions are totally free and really just an opportunity to talk shop, ask questions and get answers.

Register here:

Basically, these sessions are an opportunity to get a free weekly consultation with Cloud Posse where you can literally “ask me anything” (AMA). Since we're all engineers, this also helps us better understand the challenges our users have so we can better focus on solving the real problems you have and address the problems/gaps in our tools.

Machine Generated Transcript

That's me.

That's right.

You know these days there was pride is very private.

Conservatives want to really make it clear you know what's going on.

All right, everyone.

Let's get this show started.

Welcome to Office hours today.

It's December 2019 and my name is Eric Osterman.

I'll be leading the conversation.

I'm the CEO and founder of cloud posse where a DevOps accelerator we help companies own their infrastructure in record time by building it for you and showing you the ropes.

For those of you new to the call the format of this call is very informal.

My goal is to get your questions answered.

Feel free to unmute yourself at any time if you want to jump in and participate.

We host these calls every week will automatically post a video recording of this session to the office hours channel as well as follow up with an email.

So you can share it with your team.

I've been slacking on those emails lately though.

So being me if you don't get the message if you want to share something private just ask.

And we can temporarily suspend the recording.

With that said, let's kick this off.

I have prepared some talking points as usual.

These are things that have come up throughout the week.

We don't have to cover these things.

These are just if there are not enough questions to get answered today.

So first thing is that Martin on our team also in Germany there's somewhere I shared a pretty cool little module he's been working on, which is how you can deploy and bastion and AWS that has zero exposed ports to the outside world.

And you can you do this using the New Features of the SSL sessions manager for shell.

Also in other news, the engineers at headquarters in Moscow are rated kind of interesting drama in our industry to unfold 15 years after the company was started.

EU plans to deprecate helm based installations.

I'm really not liking that.

I hope this has not a trend.

This makes me think of the Java installer we've had to put up with for a couple of decades here.

And lastly, I want to cover something that we briefly touched on last week with the announcement that you know the official help chart repositories are going away.

What is a alternative pattern that you can use.

That doesn't mean you have to host your own chart registry.

And I want to talk about that too.

But before we cover this in more detail.

Anybody have any questions, we can get it beyond Terraform Kubernetes DevOps and general job search.

Anything interesting or an idea.

This is Project Adam cruise towards the end.

I'd like to talk a little bit about Paul request process and how to be able to get something of a bug fixes and changes contributed to some of the cloud policy.

OK Yeah, it sure will.

We can jump in and talk about that right away.

First thing.

Have you already opened something in your app.

You're frustrated that we are moving fast enough.

That happens too sometimes because we had a lot of freedom pull requests.

Yeah, I have.

I have to open right now.

And I just kind of care like what the expectations are.

For how long it should take to get reviewed or you know how long.

I should be patient and don't feel bashful pinging us in Slack to prioritize.

It's kind of like if we follow.

Sometimes the path of least resistance.

So if you nag us on the slack team.

I'll have Andre or myself or you or one of the guys look over your request and we'll try and expedite it.

So for starters can you post your PR to the office hours channel and I'll call on call attention to it.

Absolutely You can review that one with a one page deal.

If anybody afterwards wants to help me troubleshoot us certain that I'm doing GitLab install and GitLab on a career that is cluster and cert managers give me a pain in the ass.

So I can send you guys some cookies if you want to talk about it later anyway.

OK I read an article yesterday regarding alphabets move towards trying to be number one or number two in the code space.

Yeah with their timeline.

I seem to actually have a response that they may end up just putting it all together.

I don't think it was good PR for Google.

Now on the iCloud product.

Yeah, it's great.

They want to be number one or number two.

But Google has a horrible track record of it.

And thank you for bringing this up.

This should totally be a talking point.

But yeah Google has a horrible track record as it comes to pulling out of products.

We were using Google higher for example, and less than a year after we signed up and got everything in there.

They pulled the plug on that like so many other things.

So I would not put it past them to just say, hey, we're exiting the cloud business.

Sorry we did become number one or two.

We only want to be number one or two.

That's your fault. You should have made us number one or two.

Yeah like to just pull out of GCP with them.

I don't know.

But I mean so.

So very distant.

I said I read the article or something you said like their cloud business maybe generate somewhere in the $8 billion dollar per year range.

When Amazon does that per quarter something like that.

I mean, yeah, there's a big gap.

But it's still.

I mean, does that make it not even worth it for them.

That's amazing.

Yeah because I see what I see where Google does some things very well and AWS does some things very well as well.

But it's still a part of the market.

And we can actually grow it was they can actually do other things even better than Amazon.

I think what Amazon has been doing really is like the Facebook model that be looking at trends to what people are doing.

And then build a service around it.

And then they push that out as a repeat each time.

Yeah, I would love to see the Amazon formula.

I bet they have something pretty well-defined internally about how they identify assess scope ballot build requirements prototype.

I mean, it's a machine inside of Amazon how they're pumping out these services.

Yeah, that's your point, I mean, I think a lot of certain Google services.

I'm envious of the UX is a lot better.

The performance is a lot faster.

I mean GKE compared to cars.

I mean night and day difference in terms of usability and power and updates.

Yeah, I must say even like after filing a couple GTM b bucks that was super quick to fix them like I was using Google functions on the product.

I fixed the back line.

I don't know two weeks or something that wasn't really even important.

So it's like this relationship between it was not even a big company that was working on a school project like this that's I fix it in two weeks.

And I just feel like the creation ship is a lot better between Google and its customers, even though a lot of people say Google's sucks its customer advancement.

I think they've I think they've turned a corner on that.

I mean, Google is G Suite user and a Google Fi customer.

Their mobile offering here in the US.

And it's the best experience of any product that I have.

Also they're Google Wi-Fi.

I use that as well.

And that's been a phenomenal experience working with their customer support.

I talked to somebody within about five minutes of calling up their technical.

They know what they're talking about just not having to navigate 25 menu options sit on board for 45, 50 minutes.

That's what's great.

Yeah, I had that experience just recently with a guy at code fresh.

And I talked to a good experience.

It sounds like.

Yeah, you know, the get set.

And you know you get you get shown a demo and you know sometimes it's somebody who's just been trained how to do the demo and you try to deviate at all.

And they're like, I don't know.

I don't know what I'm doing.

But this guy was like, you know, all kinds of stuff you know.

You know and showed me.

I was asking all kinds of questions.

And he was going around and showing me different things.

And we were totally off script and I really appreciated that.

You know this week, we had a technical guy in front of me that it's very cool.

Do you know who it was by any chance.

I do come back to that.

Oh, yeah.

I'm glad those are a positive.

This is place I can.

If anybody is interested.

I can stick to how things happen at Google internally.

Having worked there for four years.

They're extremely good at doing stuff in scale.

In other words, things that they do in scale.

They take very seriously.

So if what you have if you need them to do something that might be somehow personal or unique to you you're screwed.

Just never going to happen.

Yeah, but if you need.

If you need something that is, in a sense part of their existing workflow you know they stay up late at night trying to make that happen as quickly as possible because they want to reduce the friction.

Yeah Yeah.

So basically, if they see that this affects 10 million people, then they will go do it.

Yes And I'll talk a little bit about what they can get away with that.

The other thing is that they have two different.

There's basically like two different categories of services in terms of support in terms of internal IQ support or SRT support.

There's the stuff that's funded or I should say the stuff that has what they call a revenue stream.

And so there's a billing idea for it.

And those things, then get different tiers of support internally depending on how much they go for.

And then there's everything else.

So basically, if in their internal accounting, they don't have a revenue stream associated with a service.

It could disappear at any minute and they wouldn't give a rat's ass.

Nobody would try.

That's harsh.

It's very harsh.

Yeah, it's totally.

Yeah And the other thing is the way they can get away with it is they do a lot of blue green deployments.

And that's basically something that I would love to bring up in this group at some point, maybe not today, but how we can go about establishing quality gates and running canary mills.

So at any given time, there will be multiple versions of female running at once.

For example, and the versions might be like unbelievably similar to each other, but they then start to collect telemetry.

You know they compare the metrics of the new whatever.

Well, this is the feature flagging is for right like launch darkly or Flagler or flipped except well Yeah, except it's more about having multiple.

In other words feature flagging.

Yes except you're comparing the flagged instances, the flag pods with anon flag pods and they're all running simultaneously.

Right I was to the beauty with things like flag or is that that's flamer trees built in.

So you can see how that's performing for you.

And then you can tie that in with Prometheus and all your other reporting dashboards.

So I guess what you're monitoring or measuring is going to depend on the product, but Yeah, that because the tooling for that is there.

I guess there's no good demos for how you should do it.

Yeah And then I think the ultimate the ultimate secret.

I mean, it's not really a secret.

It's just that they have the discipline of making sure that whenever they design something they include the diagnostic criteria, as my guide.

So if you're writing a piece of software.

And it's going to do something, then something in the world should change as a result of that piece of software.

That's a good point.

Viner you have to say this is going to affect these metrics.

And these are the thresholds that are considered good.

And then so like zero might be bad.

For example, let's say you want to tell people in the spec 0 is bad because then that automatically gets plugged into all the things that could be bad.

Yeah, I totally agree in theory with that.

I think few teams have the rigor to be able to actually carry that out.

And the and the foundation to truly report on that, well ability you're saying I like the idea that before you make a decision, you decide up front kind of what are some of your expectations that this is going to do.

And then by deploying it and made seeing that change happen.

Did it meet or exceed or not at all or did it meet or exceed your expectations or was something in kind of a similar vein that my team has been going through is that we started saying because we get people that say, hey, can you create this little app for us you know as like internal research and development type thing.

But you know we don't want you to spend a lot of time and money on it.

So don't do a pipeline don't do any tests just do it.

And so what we started saying was no if you don't want a pipeline if you don't want any tests.

It's not important enough for you.

We're not going to do it.

You so we have this new philosophy that you know everything.

It's pipeline.

Yeah nothing doesn't get a pipeline.

Pipeline driven development.

Yeah And it's so the this video that I watched from Uncle Bob.

Everybody know uncle Bob Martonne from La.

He's one of the founders of agile agile veterans.

He did a video and he started talking about how you can't go to your business and ask, can I write tests.

Can I build a pipeline.

Because what you're doing there is you're trying to shift the risk to them.

Yeah, they don't know that stuff you know, that's you.

You know that stuff.

It's your professional you know risk to take.

So he said they're just always going to say no, you know.

So you have to shift that mindset.

And you take on the risk of you know saying, I need to write to I will write tests you know as part of my professional ethics you know or whatever for this is the way I operate.

Yeah, that was really powerful.

That is powerful.

Thank you for sharing that.

I like that you have a link to the video shot and please share that.

Yeah, I will share it.

And just to add some context in context.

But add some reaction to what you're saying because we all know and I've been there.

How if you didn't do that and you go along with doing it.

Next thing you know, you get a notification.

So we're going live with the service next week.

And now it's your responsibility to manage the uptime yet none of the legwork was done to make sure that it was a reliable service from the get go.

So you called.

So Anybody have any.

As anybody first of all played around with the Systems Manager shall I. We haven't done it.

We use mostly teleport.

So I'm curious if anybody's using the systems in your shell what their experience has been.

I have not.

But I do want to hear a little bit more about before from when you use it well and managed by the authentication.

Gotcha Yeah.

We can certainly talk about that a little bit.

I have somewhere in my work a my if you to go to a climate policy startups.

So here, I think under screenshots here.

I have some screenshots of what the architecture looks like.

Portillo port, which is helpful because it is a legit like enterprise grade application in terms of complexity and architecture.

Not because they're trying to be difficult. But because it's solving a difficult problem.

So we just gotta let this load here for a second.

It's going to be a little slower since I'm logged in.

And this is not cached session.

In the meantime, for everyone else teleport is a product by a company called gravitational grab.

They actually just raised like 30 million.

So they're doing pretty well to go to GitHub dot coms large gravitational slash teleport.

It's a open source or open, core project.

Most of the functionality is in the open source version.

But things like single sign on with Oct or anything other than GitHub requires a license to do that.

So let's see this loaded is screenshots.

So teleport.

So yeah.

So just like the cool thing with teleport is you can replay sessions like with full on YouTube style replay so that's what you see right here.

This is a replay of the session.

But the architecture.

Here we go.

See this is big enough can you guys see this.

Maybe we need to make the bigger.

I think you were good before.

All right.

So there are these three components.

Basically the auto serve as the proxy service.

And then the node service.

So the node service is this service over here.

And what you do is you deploy on all of your servers.

You deploy this node service and that kind of replaces ss.

So technically it can work together with OpenSSH age or it can replace open SSH.

Then you have the proxy service.

So the proxy service here is deployed as you can see multiple times.

However, it works like a tiered architecture.

So basically, the proxy service is your modern day bastion in teleport so you expose one of your proxies like this, and you set that behind and you'll be with TLS and your ACM certs like tell that I can't 10x that I own in this case.

And that connects you.

So when you connect publicly either using the command line tool using Clyde tool called DSH or you can connect over the web like this, you're actually connecting to the proxy service here.

Then there's that.

So then you have say your clusters.

And you have multiple clusters.

And this would be an example of one cluster here.

But let's say you have five or 10 or whatever clusters all of these deploy the same stack of services that you have they have an auto service they have the proxy service.

And then they have the node service running on all the nodes you typically run like one or two proxy services for AJ one or two off services for AJ in each cluster and then a node service on each node.

So what's going on here is that when these services come online the proxy service comes online and tunnels out of your cluster to your centralized proxy service.

This ensures that all of your clusters don't need to be publicly accessible because they tunnel out through the firewall.

Then you have your order service.

So he uses the TKI infrastructure where each auth service has its own certificate that's been signed by the CIA central caa and then that establishes the trust.

So that this Ott service here can handle the issuing the short lived certificates for clients on age.

So let's see here.

I know.

So I know enough to be dangerous.

I'm not deeply technical and teleport because it is a sophisticated product.

Other things to point out here is that each service here writes to a Dynamo table.

It's events.

So you can have an event stream of what's going on there.

And then you can tie that in with lambdas like Lan does that send slack notifications and then all the sessions.

These are your TTY sessions.

Those shipped out to an encrypted S3 bucket.

So a lot of companies have tried to h logging and they will send those logs to service like single logic, not to single anyone out here.

And that's cool from like one perspective discoverability having it all in one place.

But it's not really effective if people are using TTY sessions like going into them more using top or running anything that does like ANSI codes to reset the cursor position and all that stuff, because you're not going to see what's going on when you look in those regular log files.

That's why you want YouTube's out playback and that's why they send a binary log of the session.

What's happening to this encrypted S3 bucket.

So deploying this.

So there's the way we've done it and the official way to deploy teleport and there are pros and cons with both strategy.

So let me talk about the official way of deploying teleport.

So the official way of deploying teleport is that you should not deploy it inside of Kubernetes.

It should be deployed at a layer below your container management platform.

So basically on the coast.

And that means like deploying it.

So it runs under System.

And the reason for that is that this is how you treat issues that could be even affecting Kubernetes itself.

So by deploying a layer underneath Kubernetes gives you the escape hatch you need to try things going wrong with your cluster.

For example.

The downside is that introduces more traditional config management strategies or dependency on that or you need to roll your own images with Jetpack or something that have those binaries built in.

We've kind of taken out a compromise to this where I could possibly deploy this as a health chart and that allows us to use the same pipelines and tooling that we have for everything else we do.

And we deploy the odd service that way the proxy service that way.

Using health charts.

And that also means that Kubernetes can manage the uptime and monitoring of those services and means we can leverage Prometheus with all the built in monitoring we already have in Kubernetes without adding anything custom.

So there's a lot of other benefits to deploying this.

Also inside if you go to cloud passes get up.

So GitHub slash cloud posse slash Helen files, you will find our helm file deploying teleport under Kubernetes.

But keep in mind that teleport like so many other things don't just run self-contained under Kubernetes they have banking services that they need.

And that's why if you go to GitHub cloud posse and search for the keyword teleport under our repos you'll find the Terraform module that has the backing services teleport so teleport.

So Yeah.

This is the module that you might be interested in if you're going to deploy this.

So this deploys the S3 bucket following best practices in the DynamoDB table following best practices.

So that makes everything else capable of being deployed basically inside of communities.

Let's see what else are Docker image.

Don't use this doctored image.

It's not been updated.

As you can see share this to.

Were there any other specific things you wanted to see.

So that was just a high level over.

I can go deeper or more technical if there's something in particular, you were hoping to see on that.

So you would major one from the start.

Yes Yes.

Yes Yes.

OK Cool.

So So yeah, we used that we deployed with the enterprise offering the enterprise offering enables G Suite integration for example, or Oct integration and then you Mac, then you use.

Let's see here.

How easy this is to show.

If we go to.

So we have our own charts for a teleport that we manage.

These are different from the official charts.

Jupiter there is more than that yet teleport enterprise off teleport enterprise proxy.

So in here, you can see that we provide some example.

So I'm making the assumption that one side provide access to a user.

I can also restrict this restrict them to specific resources.

Yeah So you map basically the role that is passed through the sample or so.

So when they authenticate it passes the role.

And then you can map that role to a role inside a teleport.

You can configure all the roles you want.

So then teleport you can configure like that.

This bill allows root access to this cluster or this role allows root access to all clusters.

Now you couldn't you couldn't make it more fine grained like read only cats that read only as a concept that only exists inside of the operating system itself.

And this is h doesn't govern that.

So you can control what it.

This is h account, they can.

I'm trying to remember C I have not been doing the deployment of teleport.

This is a Jeremy on our team.

He is the expert itself.

So I'm going to have to plead ignorance here on how that role mapping is setup.

I believe it's deployed through a config map somewhere.

Here the files would that be helpful.

All right.

Have you given more thought how you plan to deploy teleport.

Not yet.

I'm just tired.

OK, here we go.

So sort of playing it with Helen file.

And then we define these templates here, which are the configuration formats of the chart was similar to t descriptor.

Guess we don't have any examples of those in the public thing here.

Yeah Sorry.

No, we don't have a well documented explanation of how that works right now.

But if Yeah, I can that name Jeremy for some more information if it comes up.

Cool any other questions.

Sorry Dale I can answer that question precisely to get you.

Many other questions.

I mean, that was it regarding that.

Yeah or anyone else.

Any questions.

See you.

What were some other talking points.

Anyone have any commentary on the whole engine X-ray.

That happened in Moscow.

They sent out an email.

I heard it thing kind of worked out OK.

Yeah, good.

Yeah, there was an email that Nginx sent out to all their customers or something.

OK Yeah.

OK That's cool.

Yeah, I got to look that up, I'm sure they were like full disclosure.

This is what happened.

You know.

But nobody was arrested.

Bubble blah.

OK So nobody was arrested.

I'm just thinking two guys very well that in the original reporting it said they were brought in like you founders of engine x were brought in.

That's what it said in the original.

But you know maybe that was just hearsay or secondhand and not confirmed now.

I found it along the tweet creflo given how much of the internet is underpinned by this.

I was worried about it.

I'll post a tweet that I found that talks that that post a screenshot of the email.

OK, now we always strive to be transparent with you or valued customers and we're committed to keeping you updated.

OK enforcement.

All right.

Yes, they were just brought in for questioning the founders.

They were not technically arrested.

They were brought in for intimidation.

All right.

Could something like this happen.

Yes What was that code.

Something like this happened in the US like because of copyright infringement.

People get arrested.

I mean, if they want to they could do whatever the hell they want.

You know what.

What did happen in the US.

No probably not.

But the Patriot Act and stuff.

I mean, they could lock you away.

OK The good thing.

Again And yeah.

And if it's a Hollywood movie they might bust down your door with a SWAT team because you might be a violent offender or something.

I don't know.

I mean, this whole thing reminded me of the pyruvate right in 2008.

I think it was when they raided the data center.

Was it later.

I think 2007 2008.

So yeah, I mean, it's a you it could happen.

I think.

Like people could get arrested for it.

Remember Kim and the raid on his mansion in New Zealand.

That was pretty insane.

So if one paramilitary whatever planning Margaret Mead did the same thing with his mom as well.

They actually believed.

Oh, they did Kim Dotcom mom.

Yeah well made it personal.

Well, so I don't want any of you using a CEO we've started using it with a customer.

That project was delayed because we kept on waiting for the helm chart to become more stable.

And we also saw that they started a new escrow installer for Hellman project repo.

But then Ryan in our community called out this thing to my attention, which kind of bummed me out.

I don't know if it's true or not.

They said that they're going to be deprecating helm the helm installation approach.

So I get it.

As a developer kind of why they're doing it you know they're frustrated with some of those limitations.

But I don't like it in terms of the trend at this.

This makes me feel like the modern day Java installer where you can't just apt get install the regular Java you have to go sign up.

Give your email download a jar curl it or type it into bash agreed to some terms of service.

All this other stuff.

Is this the future is you.

I hope not.

Well, that's not even the case anymore.

I mean, I could just go say you know brew install open data.

No Yeah well good.

Open it.

Right but that's not the son version.

Well, so I'm hoping that there'll be an about face on this decision because I do like installing everything in a consistent manner using it.

I don't have a huge problem with it.

If it's providing other like if to do it with helm required more than just helm install it still if it required more than that, then it makes sense to wrap it with something so that it can just be you know Estela seitel installer.

The CEO.

Right I mean, you want to make it as easy as possible for your users.

Yes, yes, and yes, you do want to make it as easy as possible for your users.

But sometimes these are at odds with each other.

We had this conversation last week about home and how I think you actually brought up that you are disappointed that a lot of the home charts are insecure by design or insecure a box.

And I think their goal there is to make them as easily install as possible.

Does odds that make as operationalized production eyes as possible.

Oh, yeah.

I mean that the people behind this trio in particular tend to buck the norms anyway, so like another perfect example is you will not find in this video channel in the Kubernetes slack.

They have their own.

So that's just the way they operate.

At the same time fluently and blinkered and all the others mean, that's an issue and specific.

Yeah, I want to hear from them.

For me, the only issues that I see is like how do you make this replicated all across multiple clusters.

Like for example, you have a Def Con so you want to cancel this show on it.

I mean, calling a CEO, I command works.

Is it like some way to go for now.

Exactly I mean, it's basically like using the aid of US clock command right.

And we all know.

But we're not used to it obvious like manage to define our infrastructure right.

That's why we have tools like Terraform.

That's why we have tools like help.

So how do we I guess I'll be OK with it.

But then they need to provide a declared way for me to describe the state of my CEO and make it a you know if they can make it like it still apply and still destroy and Yeah, that, then I guess I'll be OK with it.

That said, I just don't feel right.

It doesn't feel right.

Yeah something doesn't feel right.

And so you know, we've been a big proponent of health file and health file is a terrible one.

But that's why I'm so happy today that there's a health file provider.

Now that's helping us integrate health file.

It's our chair from workflow.

So does that mean, we're going to need to have a CEO provider for Terraform maybe or something like that.

I mean, I would like to know the reasoning was.

I did it.

I mean, it's something that really is like I cannot solve flip with home and home sites.

We don't want to solve this case.

I mean, the only thing that I could think about as a CFD that I kind of an issue.

Yes, dear these are kind of poorly supported.

Underhill due to not if somebody could explain better than me that go for it.

It's basically like the race condition between the CRT getting deployed with the containers coming online and waiting for those resources to be functional isn't supported by health.

So you get these errors that like this resource is doesn't exist.

Well, it's really just in the process being created.

I think that's the gist of it.

I definitely encountered this problem.

We the way we worked around it is.

We just I think I'll hook exactly you know on file hooks with I think calling kubectl apply right or on the map.

Yeah, there's just some issues when you try to have a resource of a specific type.

And that type doesn't exist yet or it's in the process of being registered.

There's something going on.

And it's annoying, especially with spot manager.

Yeah And there's still need to have their customers here it is.

These days.

Yes So I know I know this affects me most as well.

And I know he's been contributing some powers to fix some of these problems.

But I don't know if the root causes to have been addressed.

I thought I saw something in a change like a while ago, a few days ago.

But that.

Oh, really cool.

See if I can find it.

Yeah be sure to officers.

If you see the steel city on manifest apply for you're going to want.

Oh, nice.

OK So they are doing something along those lines.

Yeah, they'd be stupid not to.

Cool Yeah.

Well, one of the last things I wanted to cover briefly was I know many of you are using help and could be impacted by the shutdown or the eventual deprecation of the official charts repo.

There are a lot of charts out there.

And I don't necessarily want to have to figure out where their chart registry is or make sure that it works is up to date.

Very often you'll find the chart registry.

You'll find the chart.

But they haven't pushed the latest package up and you want to use it.

So we've been there.

There are a few of these plugins out there that add the get protocol support to help.

So you can just use like a get slash scheme.

This is the best one though, that I found and I forget his non GitHub name, but he is.

He's in suite ops as well.

And a hell and file user.

So this project together with helmet file works.

Literally point to projects anywhere on GitHub.

So let me see if I find it console slash in this repository.

Find an example using its Yeah.

So here's an example of the external of the chart of the chart repository we're pointing straight to a GitHub repository and pinning it to a committee.

So this combined with helm file makes it pretty awesome because you can really deploy from anywhere.

You can also deploy from the local file system.

If you want to works with help 3.

Good question.

I haven't tested it yet.

We'll be doing more home 3 stuff in our next engagement.

I've been playing with help 3 on home tile.

How is it working for you.

I am in the initial development phase.

So I haven't deployed anything, but it looks like it's working just fine.

I like it.

I've got work mind templating glinting everything's going fine Woo.

Same here.

Looks great.

OK, perfect.

And you are using him file as well.

Now Entered Yep.

Yeah, so we're working on where we're going to come up with repeatable like what we talked about last week.

We're taking the Terraform route modules approach to helm deployments as well.

So we've been asked to come up with a highly repeatable way to deploy a lab or Jenkins or center cube you know just spit them out all over the place.

And each time they get deployed they need to be fully production ready you know.

Yeah And so that's like the route modules approach where this is our route module and the route module in this case is going to be a good.

It's coming together.

Have any of you helpful users kicked the tires on the helm file provider.

Not Yeah.

We've been working with a little bit.

It'll work just fine if you're not using Terraform cloud.

But if you're using Terraform there's some challenges.

The idea is to have everything from the cluster to the apps are on the cluster all and Terraform.

So yeah, for at least everything up until like you're up through your shared services.

Not not necessarily using it for application deploys but using it for things like GitLab deployment, for example, to use Andrew's example, we don't get to use that to deploy you know like maybe cert manager to the cluster we defined the cert managers.

All that stuff.

And help file because that's how we have it today.

And then use Terraform to call that I know it's just entirely.

It's because of the dependency ordering.

So you can tie it altogether more easily.

But yes, it is layers on here a layer is it.

And I think that could be a good.

Next step that would be easily something that would build upon like those root module helm files that we're doing now.

Would just call the home file and we did something.

So I work for a government systems integrator and so the government has been for the last year or so, putting out their RFP is with tech challenges like hands on style meetings where you have to your team has to go into their office and they sit you all down.

And they hand you a piece of paper on it with 10 user stories or whatever.

And say, OK, you have six hours to go and they hand you keys to an empty AWS account, and they expect are running at, with full infrastructure you at the end.

So you have to come in with your infrastructure as code and everything all ready to go.

So we built a rebuilt like what you were talking about with Terraform and helmsman.

And it worked pretty well.

It was not very customizable it would just it would do its one thing and it did it fine.

You know.

But like if you decided, well, I don't need Jenkins.

I'm going to use something else instead too bad you're getting Jenkins.

So our work that we're doing now is kind of expanding on that into coming up with ways to say, all right.

You want to Cuba in this cluster do you want it.

You want it in the US you want it in Azure you know and then OK.

You've got your Cuba cluster do what you want in your Kubernetes cluster, you want you know GitLab you on Jenkins do you want a Server Manager Bob Barr you know and that's kind of where I have to ask you a little bit more about that project later on.

And I have a question on that as well.

I think more in general with using RBAC with eight of the S and I am we guys done that successfully and with ease Yeah.

I mean, it's right now.

You know we haven't done anything super fancy with it.

It's you know when you stand up.

Yes it.

It's you can map users or rolls to rolls inside Kubernetes.

So I can say anyone who has assumed this role in I am when they log in Kubernetes they are this role in Kubernetes.

So we have a role in I am called Kubernetes master and when they log into Kubernetes they get assigned cluster aban.

Right Yeah.

So it works fine.

Are you just managing everything on the I am side not with the it off config map or do you also have items of that conflict map as well.

Yeah the US off config map, you need to add what the real name is.

But not the use this rule that you are assuming.

No just the role.

You don't need.

I mean, you can map users.

But we think that's done like you know because you want to map the role.

Yeah And then later on.

And I am you get you map you know who is allowed to assume that role.

So Yeah.

So you can see them.

For example, in a group that actually has that rule attached to it or something like that.

Yeah OK.

Yeah but all right.

So do you then have to provide them the who config with that roll assumption in your post.

No we give them the ability to generate the kubeconfig so AWS chaos update kubeconfig will generate content with the roll associated to that user.

It's a flag.

It's a dash dash roll or an arrow.

Oh oh link it sounds good.

Yes, please.

Yeah, that's where I got a bit hung up with allowing them to just use the end of this while angel.

Yeah, we started out, we started out passing the Kip config around all over the place and quickly realized that that was a horrible way to do it.

Yeah has anyone been using the new farm gate on cars.

I haven't.

But looks interesting.

I'm not sure what to expect.

But I think we have a tear from my book for it, which prototypes and tests it.

And you can look at that.

There's a lot of limitations to using it.

And I think it's a little bit early probably to use it.

You'd like that's more the implementation.

I like that dp has done with their own GKE.

It takes care of a lot of the just management headaches and I just can start.

My applications with databases like I'm actually building my own house and I like the bigger things.

I'm going to go out you forget about or some dependencies that just get overlooked you know.

Oh, yeah.

Cause like I'm sticking with our back.

For example, I can implement it very quickly with Katie without an issue with it always is a little bit more work.

Yeah, everything's just a little harder like what Katie.

There's just a checkbox to have right.

I mean, it's that kind of stuff kinship shouldn't link with me.

I appreciate it here in the office hours.

Cool right.

Yeah here's our module and here's the first pull request.

And I Andre links to the but the you know, some of the limitations here.

But just to emphasize what those limitations are.

I just copy and pasted it in here.

So you can kind of see what some of those are.

So it doesn't support network load balancers right now apparently.

And what are some other ones.

Yeah, you can't read Damon says things that doesn't really make sense anymore.

I guess since there's no traditional concept of a node in that sense.

So what have you decided you'd like get a dog.

A deployment.

All of those.

Well, Yeah in 30.

Yeah, if you need to use data to log for data.

It's not for you, right.

It's for the people who are running temporary workloads.

It looks like somebody you could just run jobs on like.

Yeah Yeah.

No like if I were like.

So we run.

We run Jenkins and cabinet is now.

And we run.

If some miracle worker pods like I could easily see us using something like fa gate to run our and federal worker pods without having to worry about the underlying cluster because they don't need any of that stuff.

They don't need a load balancer they don't need you know they're just compute you know it's you and RAM that's all they need.

And it looks like all that far.

They can provide it.

Cool well, we're coming up, up to the end here.

Are there any other questions to get answered before we wrap things up for today.

Well, I'll take that as a no.

Hi, everyone.

Looks like we reached the end of the hour.

And that's going to wrap things up for day.

Thanks for sharing.

I learned a bunch of new things today as usual.

As always, we'll post a recording of this in the office hours channel.

Feel free to share that with your team.

See you guys.

Same time, same place next week.

All right.

See but.

Public “Office Hours” (2019-12-11)

Erik OstermanOffice Hours

1 min read

Here's the recording from our “Office Hours” session on 2019-12-11.

We hold public “Office Hours” every Wednesday at 11:30am PST to answer questions on all things DevOps/Terraform/Kubernetes/CICD related.

These “lunch & learn” style sessions are totally free and really just an opportunity to talk shop, ask questions and get answers.

Register here:

Basically, these sessions are an opportunity to get a free weekly consultation with Cloud Posse where you can literally “ask me anything” (AMA). Since we're all engineers, this also helps us better understand the challenges our users have so we can better focus on solving the real problems you have and address the problems/gaps in our tools.

Machine Generated Transcript

Let's get this show started.

Welcome to Office hours today is December 11th.

2019 and my mom's birthday.

Happy birthday mom.

My name is Eric Osterman and I'll be leading the conversation.

I'm the CEO and founder of cloud posse.

We are a DevOps accelerator.

We help startups own their infrastructure in record time by building it for you and then showing you the ropes.

For those of you new to the call the format of this call is very informal.

My goal is to get your questions answered.

Feel free to own yourself at any time you want to jump in and participate.

We'll host these calls.

Every week we automatically post a video of this recording to the office hours channel on suite ops as well as follow up with an email.

So you can share it with your team.

If you want to share something in private just ask me and we can temporarily suspend the recording.

With that said, let's get the show started.

I do want to cover some things that we were just talking about before we kicked off the call here, which is on efficient ways to track your time as a contractor.

If you're doing that type of stuff.

But also here's some other talking points.

If we run out of things to cover a public service announcement here it was AI believe that brought it up again and reminded me this morning.

The official hymn chart repositories slated for deprecation in about a year and there's a schedule announced for that which is on this link.

I want to show you our new slack archive search functionality which makes it a whole lot easier to find what you're looking for in the suite office chat we use.

I'll go live for that I want to show our new guest modules for two kinds new managed node pools for each case as a result of the announcements from reinvent and then one of the things Andrew also asked about were like what are the considerations that we take into account when thinking a CIO platform and I expanded this to be set out.

But let's just quickly jump out 2 this thing about tracking time.

What was that cube called that you were talking about.

Andrew it's called time M��ller so you get the app.

The app is free but you can setup.

I think it's got eight sides or six sides or one to three it's got eight sides so you can set up eight different eight different tracking targets whatever they are whether they're different clients Lauren Berman or no I don't.

I didn't buy the I didn't buy the physical device yet but I have.

I'm using the app oh that just lets you quickly tap on.

So I set up I set up three I set up I've already set up three targets one is my one project that I'm on and the other project they run and then the third one is overhead, which is like time that I spend that I can't charge to either project and so I can hit Start stop on any of them as I'm working through the day because I like this morning I started to work I started work today this morning it I don't know a 30 and I spent an hour just plowing through emails and I thought and I stopped myself and I'm like how am I going to charge that last hour that I spent because it was like an email from this place and then an email from this place and like I didn't know.

And so I was like, all right I've heard about these devices and stuff.

So let's look into them.

And so this app is free I'm using it now I'm thinking about buying the actual device because then you don't have to look down at your phone you just pick it up and move it to another face and it automatically sticks with your phone with Bluetooth.

The device itself is like $69 or something but you know hey it's Christmas I wonder if there's some way to integrate that then with either zap beer or harvest directly because we've been using part of this to track time.

Currently, we offer the following integrations toggle Jira harvest.

I call Google Calendar and outlook calendar cool.

And then who was just talking about walking time walking time has come up before it's me Yeah.

So I use it to basically track however my projects.

So Brooke's great but I don't have to do any serious billing.

It's just like OK most of us at a time of day on this project and had to do so.

And so.

So walking time is kind of like self-inflicted spyware for consultants.

It keeps track like what files you're editing what things you're doing so you can back that out to reconstruct your time as well as track your time at the same time.

Well for developers.

It looks pretty good.

Yeah it's as if lots of integrations I mean also into trade.

So if you're emails I think if you're all right I'm there.

But it's a cool program to just track like roughly how much time you spend.

Anyone else try walking time to know mark champ schemes and using that.

Cool well let's shift the focus back to DevOps first I want to see anybody have any questions related to Terraform communities home et cetera.

Well when we jump right into your question, Andrew then you had asked me just before we kick this off kind of what are some considerations that we took into account when evaluating CCD platform and I just jotted down some notes before the call.

So I would forget it.

But before I talk about this I just want to say that when we started working it was like four years ago and yes there was a rich ecosystem of CCD.

Then and I think that ecosystem has 10x since we started.

It feels like everyone's pet project is due inventing US ICD platform.

And that's why more than I could possibly come to evaluate today.

So I think there are a lot of more exciting options available today than when we started looking at things.

And here are some considerations.

I think are basically requirements today.

So everyone can see my screen here one big thing I think is approval steps since most companies still don't feel 100% confident with continuous deployment to production for example, having a manual gate there.

Ideally with some level of role based access controls who can approve it would be great.

Now since we live and die and Slack having that integrated it's a slack to me feels like a natural requirement as well because I just want to whisk it.

I just want to get a team.

When my approval is required kind of like pull reminders does for pull requests shared secrets.

This is like a Biggie for me and one of my biggest complaints about like GitHub Actions is that there's no concept of shared secrets right now.

So when you add like 300 projects like cloud policy and you have to have integration secrets that are reused across all of them.

How do you manage that.

If you have to update each one.

So we end up having to write scripts to manage the scripts that manage the scripts the of.

So scripting basically updating your shared secrets across your GitHub repositories nutcase.

I would rather have an organizational thing where maybe teams under GitHub could have shared secrets.

I think would be a pretty cool way of doing it.

Easy personalization.

I think you get this pretty much out of the box these days when we started circle C I for example, didn't have that.

Now they do circle to other things like easy integration with Kubernetes.

I don't believe for example circle s.I. has this if you're a big time user communities.

I don't want to spend half my pipeline just setting up the connection to the Kubernetes cluster.

I like that just to be straightforward.

So this was one of the pros why we selected code fresh code fresh makes that really easy because it has a bunch of turnkey integrations to the most common systems we'll use.

What else container back steps again.

When we started this was not the standard.

Now this is the standard.

Every CI/CD pipeline basically use the steps backed by containers.

So I think that's fair.

Supporting web hook events from pull requests originating from untrusted forks.

This is a Biggie.

If you do a lot of open source how do you handle this securely.

And the reality is you can't get by like Travis c.I. will run the pipeline if you don't have secrets in it.

But these days you can't do anything without secrets so that's not a fair restriction.

So there needs to be a secure way to do that.

Typically, it's done these days using chat ops where authorized individuals can comment on the pull request and then that triggers some events.

And in the act of doing that will actually temporarily give that pipeline access to secrets that could technically be expo traded but it's a necessary risk to take.

And just set one of those up and one of my open source projects and it was a frickin' breeze.

Q did you do that with code for sure.

Right Yeah.

So Yeah it should be a breeze.

And technically that is a breeze in code fresh.

I will say that I've had a lot of issues.

But I think that's because our account has been grandfathered in through so many upgrades that are our something's wrong with our pipeline.

So we constantly have to reset our web hooks and stuff.

But Yeah, it is technically very easy in code for us to do that.

And then you just look for a slash command or something in your trigger.

Yeah let's call other nation's natural segue into this chat UPS our request.

So like I said there if you trigger everything you want to check on a pull request automatically your you'll quickly fill up your pipelines your integration tests are going to take forever.

So I think being able to do a conditional type stuff on labels and comments it is a requirement from my system.

It should make it easy to discover all open PR so it's easy to trigger this is why things like so Jenkins does this right out of the box.

But we're not out of the box but with what.

I forget what the plug-in is for Jenkins that does that.

You're using that though right.

Get a plug-in.

Yeah Yeah Yeah.

Couldn't fish had this but they just deprecated that view.

So you can no longer see all of them pull requests and re trigger the pipelines for that.

I think this is a requirement because Paul requests have additional metadata and that noted that you can't sit you can't easily simulate that just by triggering the c.I. build on a branch because you lack all that other metadata about the pull request.

So a way to trigger pull requests that way to trigger builds or jobs from all requests is a requirement manually in addition to automatically should support remote debugging.

So this has been one thing like the road for us, for example, that hasn't been a possibility until recently.

Now they've added support for that.

I here really good things about it.

Basically you want to be able to remote exact into any step of your pipeline for triaging because God knows soil tedious to debug things if the only way is to rerun the entire pipeline from beginning to end and not be able to jump in or look what's going on inside of that container circle circles has been on me about that for a very long time.

There's been tricks we've had to use like teammate to get around that in the past.

So making sure that that's possible.

Question for those of you using Jenkins as Jenkins support that with any plugins today.

Now you may be able to reach into a container that you get into a step in a pipeline.

No it's weird I get it.

I'm sorry.

Oh no you go ahead sorry.

What I've usually done in the past.

So right now, I've got a I've got Dr. builds and I'm using the multiple branch pipeline from Jenkins.

Yeah you usually do as I try to adjust if I'm doing multi container builds I'll try and keep it in a format that I can easily recreate it locally.

So you just try to recreate the same bug then money into Jenkins.

And I haven't had many issues there.

There are a couple of nuances with the chicken on plugins but as long as you don't update Jenkins you're OK which isn't always ideal but it's kind of sad reality I guess in open source.

You bring up but you jog my memory for another thing should support local execution of pipelines for debugging.

So this is pretty cool when you want to iterate quickly and be able to trash things is just to be able to run that pipeline through locally code fresh has had support for this for some time now.

I don't know about circle or Travis.

I don't get it.

Actions you can run locally or at least people have come up with workarounds for that.

Jenkins does that support local pipeline executions.

I would run a local instance of Jenkins to do it.

I've you laugh but I've done that and it works fine.

I know it's just I get it and it works.

Given our demographic of who we are and what we do.

But I wouldn't expect my developers to rattle like Dr. Campos up.

Jenkins and then I the integrations for that and then I guess there's I guess we could perhaps elements Della Jenkins you're not running local Cuban edits is not enough.

That's the good conversation to have.

I would love to debate that with anybody else who is and learn from it.

OK Yeah.

I've actually seen a GitHub action to run a local company just inside of GitHub action which model that will give us tonight.

But we're not doing ridiculous in a flattering way.

I mean that's somebody I spent a lot of time around.

Yeah derailing the conversation but we're not doing it yet but we plan on doing using kind k in yen which is Cuban using Docker in order to test.

Right now we are developing reusable helm files like similar to have cloud plus he has a bunch of reusable files.

Obviously we're like our mono chart like a reusable chart or a reusable helm files and files.

And we need to test them.

So our first iteration is just going to be you know smoke test it home file apply and then wait and then help file destroy you know it just if those work successfully then we're going to call the test.

Yeah successful.

So right now we're just going to deploy into our Kubernetes cluster, but we don't want to dirty the cluster with that stuff.

So we're going to use kind always should be really cool for that kind of show before all this is where things like spiral out of control gets worse standard charts it's true but like what if you had a chart that wanted to test external DNS and then you need the you need the iron rule for that.

You know there are limitations.

The charts for the charts we're working on right now are just services like we're working on a GitLab and a Jenkins panicky cook has any links to the most common thing that we end up needing to use Terraform for in combination with like Helm is for the backing services of chart dependencies of which roles or the most common.

Have any of you seen anything that auto provisions roles somehow as part of Kubernetes like a C or D for animals you can use to Terraform Cubans for better I think and Atlantis to do that.

But what about it.

Their form could be exactly the one by rancher.

I guess there's a few now.

Yeah I'm I'm not doing that.

I don't think I would work because of the debug implications of c or D. I think it's a beautiful beautiful in principle what I want is a better way to choose visually.

Basically what I think I would want is kind of like a circle Travis code fresh slash UI kind of for looking at the CRT is executing.

So there's like a window into that system specifically for this year.

This link relates to flux as well.

I haven't used flex firsthand but my understanding is it's also suffers from some of this visibility to debugging things when things go everything is great when it works but like when you're developing things never work.

So how do you do that.

Just quickly finish up the remaining things on this bucket list of things for conservations risky platforms making it easy to tag multiple versions of a Dockery image.

It sounds obvious or trivial but I've seen systems which literally require you to rebuild that image for every tag you want to have.

So being able to add in tags to an image I think is valuable as well.

For example, we always like to tag the commit shop of every image.

In addition to perhaps the release shop.

Not really shot the release tag.

All right.

Moving on let's support pipelines is code.

Basically the ability to declare your pipelines alongside your code itself and then ideally, the ability for whatever system you're using to auto discover those pipelines.

So GitHub Actions is a prime example of that where you don't need to click around anywhere to get those actions it's working.

Let's see should support a library of pipeline or pipeline steps.

So this has also become very popular these days.

Code for ash has their steps library get up actions I think.

Does this really well.

What platform is it that has orbs as that circle you know has orbs.

So that concept I think, is really important.

So related to this and there's few of you on the call using Jenkins.

I know there's.

I know this is possible in Jenkins as well but there's a lot of my senses I haven't heard anybody really talk it up or a lot of people dislike having this central library of like declarative pipelines within Jenkins and that it leads to management or software on the lifecycle challenges and instability.

Is that your feelings as well.

If so can you elaborate on that and what.

Why does Jenkins fail when circle C I put fresh types seed on this so I'll answer that more generally.

Tools that let you do whatever the hell you want.

Fail because people do whatever the hell they want.

I like tools that provide an opinionated way to do something because then it's like this is the correct way to do it.

So do it this way people do.

Jenkins shared libraries in lots of different ways.

Some of them work better than others.

So there is a way if an organization lays down the law on how to do it and stick to that it can work well just like works.

So like I actually just posted a link to some of our job yes.

Code for ethics.

Because a lot of our Jenkins is open source because we have alternate media using our Jenkins please.

Look at it.

But we tried to create guidelines in this is like create pipeline so a lot of it.

Jenkins job yourself lots of Groovy.

We're all soft offers.

You get a lot of let's over abstract our job yourselves so we input classes in Groovy which is super fun.

One of the big issues that we found is before my time here is that it's just like a whole lot of code.

And people like deviate from those.

And as the standards change over time you end up with series a through Z evolutions of it.

Yeah it's hard to keep everything all together.

And that prompted some of our assessments.

When I look I looked at code fresh I kind of rolled it out a little bit.

But we have like an hour ago and a junk c.d. and a Jenkins X 50 halfway spun up and I'm curious because I push my team on being prescriptive.

That's a good thing.

Jenkins X is so prescriptive that you can do it only one way versus Argo Argo underneath Argo.

You can do whatever the heck you want it's an arbitrary code execution job platform.

But Argo seat is a little bit descriptive and only does the c.d. portion of it.

So you guys how do you pick the right amount of prescription.

Example I'll always give is angular versus react no angular has a.

This is the way it should work.

You know it's much more prescriptive react is more do it however you want to do it you know you want to do it in just vanilla JavaScript great.

Do what you want to do it in typescript great.

Do it.

That's the example I usually give.

Not to say that either angular or reactor better than the other.

I'm not saying that but that's the example you should be using.

We as a thread go city and Jenkins back and forth.

And they just announce their deprecating my not that great they just announced that community would be taking over maintenance of the project.

So whatever.

Yeah and we have there's of course the library in Python that Alex is maintaining that's a pile of Python scripts to manage pipelines inside of that you can implement a yellow parser inside of that Python.

So we've had pipelines for a long time.

There's much better ways to do what we did.

But at the time that was what was a good option.

But I'm looking at Argo how you do everything in it manifesto basically.

I'm liking that that format but I'm seeing limitations because Argo city is much more of a narrow focus.

So for us to use it we're going to have to plug and play like six different components instead of with Jenkins X squared, we would have everything all in one.

We have nothing written in angular and everything written in React so I guess we've made the decision that we prefer to have flexibility, but from that analogy but Yeah I'm trying to figure out the right pieces because there's a several parts here considerations that I don't have answers for in our city.

And so it's a very timely discussion because the bank has made sure this was last week.

So I would love to follow up on this.

If you join us in a subsequent office hours on what you find because Argo has been in our kind of evaluation bucket for technologies as well.

One of the things I find like the approval steps I don't find in a lot of these things is a easy thing to implement.

OK Yeah.

Integration was slack.

Pretty much everything does that these days but in varying degrees.

I would like to say I must integrate with slack and have control over the content of the messages.

Basically there's a lot of systems they'll integrate with slack but they just spew generic messages that are kind of irrelevant for half the people.

So being able to target these messages to groups or teams and then also customize what shows up in the messages I think is something important.

And then support get up deployment notifications.

Now we are a get up shop.

We use it up extensively but I get how it has a special dialogue now where you can see where a request has been deployed.

Tying into that somehow I think is a nice thing.

And then lastly on pricing if you're going for a SaaS solution I really think that the solution should be fair to smaller companies not just enterprises support a standard price per user model and unlimited builds.

Basically the product shouldn't limit how much you want to use it and it should provide a reasonable escape hatch for going over those overages without forcing you into an enterprise locked in so that that's my list.

Any other considerations.

I overlooked that since we have a bunch of people here and a lot of those seating experienced that's a show.

Well Yeah it's a lot to ask about medication year well there once that when coach rush doesn't do so hot out because they get behind the wheel only with enterprise enterprise tax enterprise.

Yeah there is a perfect plug for the B SSL wall of shame website again.

What is it what was that.

It's the soda tax or I think Salvia whatever does someone of you know the tool CI/CD de tool that allows you to include the text of the approval step.

Like for example a default see how I'm charged or the terror from plan or something.

I really would love to f but I thought it was part of the question one more time I just did it here.

I would love to have a tool that allows me to see the approval as a terror phone plan in the approvals that like include that text.

Oh Yeah Yeah I get it.

Yeah that would be cool.

So like it slacks to the chair for most simple channel and then there is a approve or reject kind of button below that that'll be really good.

Yeah for example.

I don't mean bit apartments do this really well because they basically collapse below each other and then you take a proof.

Basically you see the step from before that.

So basically you have some contacts in the eyes.

I side one sexy eye it doesn't really badly because if you want to improve you have to do it from a different view than the actual one of the top, which I really, really dislike because let's be honest there's a lot of people that don't really look at their circle see eye to see a single step and just approve because I have to so we have a we have an implementation that we did by hand.

This was some time ago for a code fresh demo.

What's cool about this implementation that we did by hand is it's just using custom steps so it's you could implement this in any pipeline that lets you run containers.

And it does more or less what I think you're saying.

So this looks familiar if you're using Terraform from cloud or Atlantis.

It looks familiar but I think it's a little bit more polished.

And basically what we do is we have we're using the cloud posse GitHub commenter so cloud posse slash get commenter tool and then we have a number of templates that are in this testing that cloud posse dot Dockery reposts if you want to borrow that but then we output this plant.

So I thought this was really clean here right.

This is I feel like it's been a perfect example of what you're asking for.

I would like this now not only as a get up comment but also as a Slack message with the buttons at the bottom.

But it suggests also we stripped out all of my screen hijacked and then and then we stripped out all that superfluous information using scenery.

Now this was done as you can see back in April before Terraform Xerox.

Well I don't know scenery as updated support for Terraform 0 to 12 yet but Yeah.

So then clean output and then just clicking on this or this takes his code fresh and I can approve it or reject it right from here OK.

Yeah that's clean.

I was thinking like how do I approve now from that Because people will in ultimate up on whatever tool you are using and don't see the context.

So you have to kind of force them to actually click on these buttons.

Yeah which I would really follow like actual feet the context of what you are proving is super important.

It is because we never check it out locally and try it.

And we need to.

And then obviously a little bit of reward at the end if your plan is successful or you apply a successful any.

Right it's beautiful.

Thanks Yeah.

Any other considerations.

I left out here in this list afterwards so you guys can use it to form your requirements.

Did you have other platforms like like OS or so this has not been a requirement for us but it's a totally reasonable expectation for others.

You should support multi OS platform.

Well Yeah it's my request is Windows central Android.

We also have AI don't know if this requirement is a good thing or a bad thing but for city who can trigger a rollback from Gucci for example.

So we have a whole bunch of our back access for all of our pipelines.

I find it less helpful than helpful.

But some people require our back.

Yeah e.g. for approval step so I mean that's somewhat related to approval steps because approval steps mean nothing if you don't have our back or a attribute based results.

Yeah I was just trying to figure out approvals steps for our city because in our case here They don't exist.

I have not figured it out but there is a diff in Argo where you can see a difference that could produce manifest files that I haven't actually used it for valuable things.

But it's the only tool that I've seen natively show like every Cuban entity object in a UI where you click on it and see the difference between different places like the like.

Well you should talk to our friend here.

Here he is.

He has something you might like.

And then also help.

Def right.

So are you guys using charts for your releases.

Yeah well while we have helped me it's broken so I looked at home file.

We have some deft deficit and things locally.

However Argo is what templating expanding out the files.

So we're not actually in the Argo PSC which is only where it's not in production right now.

You can't run defgen in production because there's no tiller.

But I'm still sorting all that.

Is where why did you mentioned file in that.

Oh because help file is basically an automation tool around helm and that is and help file requires this to function.

And what I like about it is if you're using held file you basically have a Terraform workflow for dealing with home charts.

So there is a plan phase and an apply phase with a whole file.

But it doesn't.

So I'm currently using files to document what version of what chart is installed for a non application stuff.

So for a chart museum for manager those things.

But it's not working and I haven't bothered fixing it for some that stuff.

So it's not very production ready on that but we are thinking about killing it and moving everything over to Argo or whatever we do for 3D such that you don't have to manage things in cheaper places.

Obviously that would lose some functionality that you're mentioning that the depths of the competitive stuff but trying to understand what the pros and cons of each would be.

Let me also show you one thing that I really love about using a file as part of party workflow.

So we always bring this up like the four layers of infrastructure you have a foundational infrastructure at the very bottom next one up is your platform next one is your shared services and next one is your applications and they're back end services and usually you treat each one of these independently.

So obviously how file works really well for your shared services like the for deploying all the things your platform requires.

Things like external DNS cert manager, et cetera, et cetera.

But it's also really awesome for layer 4 because in our case like so here's our example app where we showcase a lot of the release engineering type stuff for workflows that we use.

If you go into the deploy folder here is an example.

And this shows now.

So this example app has these dependencies and now how file also supports remote dependencies.

So what's cool about that going back to my earlier requirement about supporting like a shared library.

What's cool is you can create a library of services that use applications use and use version settings point to them.

So what I don't like about my example is I'm using local references here but just imagine that these can be remote and then and then inside of the releases folder the developer can define how his application is deployed.

So this is just so it gives the developer declared way to deploy their applications and it becomes even more declarative if you use something like a modern chart which is our chart here and the motto chart is basically a help chart as an interface that you can use to deploy.

99 percent of apps that you're going to be deploying on Kubernetes without needing to write a custom Helm chart.

That's enabled because when you use hell file values becomes your DSL for your declarative interface.

In this case for in your applications.

So since we're systems integrators and we can't we need to be more.

We want to be opinionated but we need to be general enough that it supports all our customers use cases we because we support quite a lot of variations of configuration in our mono chart.

But if you are you building this for your own company you don't need all these variations and you should standardize how you deploy your apps to Kubernetes and then reduce the permutations that are possible there.

Going back to Andrew's comment about if you give everyone the option to do anything, they will do everything.

So you want to eliminate that and this is how you can do that.

So you just have a policy hey you use user use Mongo chart.

Any version you want but that's cool because now you can just look at the interface that you're using of mono chart here.

In this case of version zero 12 et cetera.

So I have to go ahead.

What we want and what we have currently deployed our super different.

But if I focus on what we want and how we think we want it.

This is super useful because I haven't looked at your example I picked off one of our versions.

We Stole a bunch and simple stuff really fun.

Moved away for that tangible stuff into darker and Docker images and set up that you know Travis builds the darker image questions into darker eyebrows here.

Those are all get triggered.

And then at that point there's a new tag on the image, which would then need to go into this chart and get updated.

At first you put the chart in the application repos and we're trying to use a well to the whole formats of how you inherit chart inheritance it's kind of changing around with some of the newer stuff.

So haven't figured that out perfectly.

But do you have an example like this that has like get up set up for when an application change happens and then everything else needs to follow suit.

So yes and no so we also have a.

So here's one nice thing about what we're doing here.

How do I set this up without taking too much time.

So one of the problems I have with a lot of these pipelines that are out there is basically they wake up when they see a new image and then they deploy that image somewhere but that is like one third of the problem.

The other 2/3 is like well what's the architecture that this image gets into.

And then what's the configuration and secrets that I need to run this.

And usually those three things are all tied together.

So deploying an image when the image changes without taking into account those other two things is worthless.

Yet that's how it is demonstrated.

And so many examples out there.

So maybe it works for something people.

I just.

If someone can show me I can't get my head around it.

So our strategy is quite interesting and I call it the helmet cartridge approach.

So I want containers to work like Nintendo video game cartridges and just for the record I mean I stopped playing video games when it was the original Nintendo with Super Mario Brothers.

I so like don't don't ask me more questions about this because I quickly don't know what I'm talking about.

But as really as two days I do.

So I want a Docker containers that I plug into my cluster.

And it just works.

And that means that it basically needs to also habits architecture deployed alongside of it.

So what we've been doing and it's worked out well for the last few engagements that we've done is we bump.

You see this.

This deploy directory is in my application my microservice.

So we're shipping all of this together at the same time.

So this container when we publish it what we can do is we can actually call help file on the help file inside the container and then that deploys the container along with its architecture and image at the same time at a premium.

I don't know I feel like I derailed a little bit there answering specifically or your question there.

Adam So I asked the leading question.

So the appropriate answer.

Yeah I'm still wrapping my head around the right amount because you're right the demos that I've looked at only solid portion of my problem.

Usually the image part of the problem and figuring out how I should thread everything together when there's a million choices is like I can take away.

But I don't know if it's the right way.

So this conversation's helping for me to just think about edge cases that I haven't already thought of.

Yeah so Yeah.

So then this basically answers that.

And so our pipeline for that.

What I don't like about our example app is I did something here that was me experimenting with a way of doing this but I we're not using this particular tool right now for this.

So I had this deploy tool as part of the image that implements like a blue green strategy for the planes container implements a rolling strategy for deploying this container.

My point was more of that is just you see here we're calling Helen file as a step.

And then if you look at the pipeline.

So we ship the pipeline here the example of the pipelines that if you go to the deploy step here here you can see that we are just deploying with Helena file using the cloud tool but you don't need the flights.

Well my point is you could actually just call home file directly.

And that's what we do.

All right.

Let me digress.

Let me pause on that.

Any other questions related to anything we've talked about or something you.

I was going to mention the the the way that most home turds get implemented right now is sad about and that is you know that if you take all the defaults what gets stood up is is like insecure dev.

Oh yes.

Yeah I want to.

I want to start a revolution to flip that you know.

Yeah you forget to set some setting.

You're going to fail up into the more secure space.

You know so that if you set if you don't set any if you don't set any parameters and you take all the defaults what you're deploying is production ready.

I agree with what you're saying in principle.

It's just there's a difference between operationalizing something inaccurate is very different from getting a PRCA and the difference between victory and success and failure is often early quick wins.

So psychologically speaking practically speaking I get why these charts try and be as turnkey as possible that like look in one button.

I get this full on Prometheus monitoring cluster with Griffon and I fully integrate it to everything in wow that was easy.

But yeah operationalizing it is now going to take 80% lot longer because there's so many things and considerations.

You gotta do.

The thing is that if we optimized for the latter which is what we really need to go to production most people would fail because there's so much that has to happen to get it right and it's non-trivial like.

And that's also where you're getting a more opinionated like.

So does that mean we are out of the box require a C So when 99% of the stuff does analysis so and then if you're going to use us.

So how are you going to do us so well.

We use key code you use access somebody else uses whatever this is like the permutations explode that you can't support.

And that's what's so aspirational.

I agree practically.

I just don't see it working.

So the way.

We're coming about it for my team right now because right now we are coming up with hem files that act as we're taking we're taking the approach that you guys have come up with for and maybe you haven't come up with it, but you're using it for your Terraform route modules we're having.

We have help them find root modules where they're literally.

This is the way I want you to deploy production GitLab.

It does require SSL you know and for MVP where we're going to support one sample because that's what we've been doing at first.

We're not saying it requires key cloak.

We're just saying it requires civil rights.

So whether that symbol coming from OK or smoke coming from actor or whatever we care.

Yeah and then what we're doing is saying and this is what I love about him file and I'm learning more and more about him file I haven't used it for that long yet but we're saying we're using the environment's functionality to say the default environment is production and if you say dash dash environment equals dev then it doesn't worry about all that stuff.

Exactly Yeah.

No environments are great for that.

We see people like you.

Let's say we have humility.

I know where we're talking layer cakes here.

Layers and layers and layers.

OK that out of the way.

OK what's cool about health file and environments is environments.

Now let you as a company define the interface to deploy your helm releases, which is in a lot of ways perhaps the final layer.

If you're using help.

This is basically how can we as a company say we want to consistently deploy our apps using Yelp because helm charts have a ridiculous variation on the schema is basically a values no to at home charts have the same interface unless it's developed by your own company.

So help file by way of environments lets you know to find a consistent interface deploy all of your apps that developers can understand.

Yes there are a few downsides to this.

When you want to figure out how to implement a new toggle or whatever you've got to go five layers deep or whatever to figure out where to stick that in that's our predicament right now.

Source leverage.

The goal is like the goal is if one of my people, says helm file apply GitLab and it won't work because it's got required variables but then they add in all the required variables that add that it tells them to add you know because it'll tell them exactly what they're missing.

Once they've added all the required variables and it works it is now the exact prescriptive way that I want them to deploy GitLab which is like oh that's the holy grail right there that is beautiful.

I do like that.

I do like that especially the emphasis you put there on the required variables and that it tells you what you're missing.

For that one thing.

Remember our talking points.

We didn't get through most of them so I'll leave for next week.

One thing I just wanted to point out as it relates to health.

The official help chart repository being deprecated.

There's some interesting discussion brought up this week getting peer you pointed out a bunch of them which is like this is going to reduce the quality of a lot of charts out there because of the lack of automated testing that most people won't implement.

People have different levels of rigor as it relates to maintaining their charts producing the artifacts uploading them not clobbering previous releases.

Basically the ecosystems matter the way their signed packages up there.

There is so many things that go wrong as a result of the official chart repository going away.

Now we all know why that official chart repository sucked.

It was impossible to get a change through because of the levels of bureaucracy and the checks and everything else like that.

So I don't know.

There's a balancing act here.

What I want to point out is one thing is that I think a big part of the success of for example known is that it doesn't require a dedicated repository to manage it.

The fact that you can basically point at any git repository and pull that stuff down and have immediate success I think is awesome.

I think it's a problem that helm has taken this staunch approach that we got to have a managed chart repository which is basically just a glorified HP server.

But yeah why do we need to make it like there are always public source controls and it's been proven to work with NPM.

Why don't we just do something like that.

So the truth is you can using help ins it's non-standard but this is so this is what we've been using now for over a year is the home get a plug-in here that I recommend.

Where is it installed.

So there is a dependency.

This is to install a chart just from a git repo.

Yeah exactly.

We have repo it's this one here.

I'll share it.

I really dig this plugin and it build because I am so good so this plugin withheld file.

You basically have your package.

Jason for 4 Cuban edits you can do also this on the positive I just send in bitch doesn't work with versioning, but you can just specify the URL to the chart.

Which also works you are.

Well, I suppose it's like maybe trying to save Yeah well I get to the tar all but somebody is going to create that will produce that artifact.

Yeah and it works like if you have like a one to one correlation between like GitHub repositories and charts that use the tar ball you were out for the chart and maybe maybe that's actually the answer here is that we should move away from mono repos to poly repos for health charts and then we can just leverage that functionality.

But this is.

Yeah this is imputing up a tarball as a really sort of fact inside your releases.

No no you don't need it because again hub every good hug repo has an automatic Tarbell artifact so it doesn't just go that don't work with the way that would that work with the way that that.

Because when you say helm package like it packages up a turbo in a specific format.

So you would have to have maintain that format on this.

So to say, in the repository it doesn't do anything else so it just zips it up.

So right.

So long as you maintain that format basically the route of your GitHub repo needs to be where the chart became.

Yeah the same format as what it's expecting.

Yeah Yeah.

That could be cool.

I mean it's worked really well for Terraform and tier for modules.

So I think it's probably a good approach for helm charts as well even though so we didn't do that with five possibly charts because we were just emulating what Google was doing originally but now that's going away.

So I guess it's a brave new world.

Well and Google loves model repos too.

Well I don't you know people try to mimic you know their model repo approach and fail at it.

You know there are certain circumstances where it makes sense but you know what makes everything harder.

So let's see here.

So here's an example of a chart.

All you need to do is create a repo called cert like I think maybe to maybe.

So what is it is called Terraform primary provider block.

I think what we should have is probably helping cert manager chart or something or something like that should they.

I think the community should come out with a canonical repository named structure and support that.

Therefore, if you're familiar with the Terraform registry I want to refer back to the x case Casey d.

I posted a few minutes back.

Well there's Maven right.

There's the maiden maybe an object the component library that's also published a prior art in that as well.

A lot of these will have that.

I'm just going to how it works.

I was just going to share this for the terror from module registry that they're very particular on how you name your GitHub repository.

Basically Terraform provider name.

I think we need that for help.

Charts so we get some sanity in the ecosystem.

Is there a similar thing for health hub and how to get something published to help them.

Good question.

It is serious.

It helps set up so you can help people.

So that's a report that gavel in there which works kind of but the quality currently of these reprocess.

Super super super bad.

I mean I get daily notifications of people publishing the same version as the same ditches like this is a mash.

And for me, it's like how can you use that child and not be notified about it and it's still the official help at home which I find super super super risky.

But nobody can have it.

How much does not track source code.

It tracks the package repo where you pub where you push up your tar balls.

Right Yeah.

I just realized Adam I forgot to finish that thought that we were talking about earlier you wanted to see the difference between two versions.

This is what you could integrate with your we've just got to compare.

Yeah here's this is what you could integrate with your solution.

Why am I not getting a diff.

I'm not sure.

Try the other one.

Try not artificially maybe a release will spark some of CJJ.

Images or process to impact mentally.

So they're kind of a reach.

So with you just take on compared and right next to you I didn't get anything they work well we go.

So you compare any two releases of a health chart and see what's changing.

Yeah this would be you bring along something I mean this is a super useful cause like when there's the mondo repos.

There is no you know versions of individual home charts that you can really look at.

So there's a couple of discussions.

I wrote this shot maintainers so there's no enforcement that the shot has to be an open source and I don't have to disclose the source pressing you reproach approach which I find super super stupid because is the asking you or you can just like look at it.

I wrote a recommendation that we implement like files you're in the home that is h like I did with this one Yeah.

But let's see you should probably just get it or you would be interesting if you could somehow partner with hip hop.

It seems like a perfect synthesis of what I'm writing with them but the interest was kind of low.

Really I don't know.

Yeah Yeah.

I wrote a lot of messages for them but the interest seems to be Well I'm kind of there but not really.

Let's see because I would like my opinion like a rapper should be like they should be at least a central not every service like the digits that should be shown upload them but its age should be checked against.

So if you go like at least some kind of checking like it's a great package just being a store to upload them that age and it just changes that you get an alert and they don't see a line.

I agree.

Something like oh Yeah Yeah there's lots of things to do but Yeah let's see whipped cream screen.

I put it there.

What I think is kind of like what should be done but let's see how it goes.

Well all right.

Any last thoughts before we wrap things up today.

I'm starting to get interested in building stuff if anybody like has an initiative going on around that like AWS building.

Yeah all right.

Yeah we should probably dedicate had more time to cost optimization techniques in the US sometime soon.

We pay for courthouse and they have a communities thing that splits open up but we're not billing to different teams because it's more of a let's make sure we don't have a bunch of snapshots that need to get deleted and a bunch of instances that are not right sized.

But I'd also be curious how to apply billing to communities.

It's also all of this stuff to do in get us to.

Yeah Yeah.

OK go.

I'll make a note of that and cover that on our upcoming office hours next week.

All right.

Looks like we've reached the end of the hour and that wraps things up for today.

Thanks again for sharing everything.

I learned a bunch of little cool tricks today.

I'm going to go check out the time tracker thing I always learn so much.

Thank you.

Again the recording will be posted immediately afterwards in the office hours channels.

This is available.

See you guys next week same time, same place all right.