Let's get the show started.

Welcome to.

Office hours.

It's march 18, 2020.

My name is Eric Osterman and I'll be leading the conversation.

I'm the CEO and founder of cloud posse.

We are a DevOps accelerator.

We help startups own their infrastructure in record time.

By building it for you.

And then showing you the ropes.

For those of you to the call the format is very informal.

My goal is to get your questions answered.

So feel free to unmute yourself at any time you want to jump in and participate.

If you're tuning in from our podcast or YouTube channel, you can register for these live interactive sessions by going to cloud posse office hours.

We host these calls every week will automatically post a recording of this session to the office hours channel as well as follow up with an email.

So you can share it with your team.

We want to share something in private.

Just ask.

And we can temporarily suspend the recording.

So with that said, let's kick things off.

We have a couple well talking points here and another one in the Slack channel.

Again, if you haven't joined our slide team go to slack dot cloud pass econ again slack dot cloud posse and you can register and join the officers channel there.

So one of the awesome recommendations was from Brian Tye.

If everyone can share some of their work from home tips and I'd like to expand that to maybe some productivity hacks I'm sure this affects pretty much everyone on the call here today.

So I'd like to learn from that.

The other question that we had just ponying up here was how it was a very common reoccurring question.

I think it comes up almost every office hours, but we learn something new every time a peer asks he'd like to have his monitoring strategy vetted for deploying Prometheus operator.

We'll get into that.

And Dale let's see here Dale just posted something Dale what's this about to get his guys working from home.

Awesome So this is by you actually yes.

Knight OK.

OK So let's Yeah let's review that in a second.

As soon as we first do the first order of business.

Any questions today outside of these two talking points that I just brought up there.

Yeah, I have one actually.

All right.

Have you.

Has anyone done Kubernetes ingress behind the VPN.

So you can't use LetsEncrypt.

So we're looking at doing a node port service.

But everything kind of feels like a hacky thrown together mess.

Are you like in a private cloud scenario or I ate of us Governor cloud behind a VPN.

Yeah not my area of focus.

So what.

First of all, what's the problem here.

I'm not sure I am understanding the problem like, why can't you just use regular and Unix ingress in that scenario.

And what's the relationship with the VPN and the ingress.

So because I'm behind a VPN LetsEncrypt cert manager with LetsEncrypt doesn't work.

Gotcha automatic.

I can't because I can't do the automatic verification.

So I'm looking at doing a like an application load balancer and terminating TLC using a certificate from ACM and then I guess doing a instead of a load balancer type service for the Nginx ingress service doing a node port type service for the Nginx ingress and pointing the MLB to the port to the hosts on the port that end indexing grass is on the node port servers.

But it feels it doesn't it doesn't feel very clean.

I was wondering if anyone else had.

Oh, yeah, I've done that.

And this is a much better way to do it.

Can you explain you know the search again.

Yes So cert manager automatically goes up to LetsEncrypt and does the certification.

The certificate generation using LetsEncrypt and the way.

Let's encrypt the way.

Let's encrypt validates that you are the owner of the particular domain that you're trying to going to search for is cert manager deploys a pod that publishes and HTTP endpoint that LetsEncrypt then goes and looks at.

And it's got some token or whatever.

And it's called a sticky B0 1 validation.

But if I'm behind a VPN, you something from the open internet can't hit it.

Can you not use DNS everyone challenges because this only requires you basically to have a public reachable DNS record for the domain and modify that.

We don't have a public reachable DNS yes, it's almost no public.

It's almost there.

Yep Yeah, pretty much.

So I actually will try to do something similar, but some of what I was looking at.

Roger tried to run it globally but committed to the well.

Is it at that you want to use less encrypt.

No, not necessarily.

I just want ingress.

Yeah So take a look at how they do their range.

To me it was with us and start my angel.

I think they actually disable it and use something else to actually do it.

I think that may Patrick steer it in the right direction.

You're not be an external violations or requests for that check.

Oh, yeah.

And I mean, in this situation, since it is a very closed ecosystem why not have your own CIA and cert manager itself.

Can you manage your TKI for you, and you just need to trust that CIA then can you even let vault to generate a search for.

Yeah I mean, yeah your man, or your cell phone can handle that whole process and you set up basically it for cert manager when we were using Cam we set up cert manager as Aca, which would then generate the certificates for I and it's all self-contained ecosystem, then works.

And we have to sell less now let's face it.

There's no nice solution for all of this because of those constraints that you have.

But to me, the your own private dressing setting up that that sounds like that that's something I hadn't thought about setting up cert manager as a CIA that we would just have to tell everyone using it to trust.

Because Nginx ingress will create self signed certificates that are signed as you know to be daddy's fake certificate, or whatever.

And there's so difficult. Maybe you can kind of trust those.

But I think they change.

Like they don't kind of stay constant.

But every employee that ends next will be.

But right.

But if cert manager can be set up as a stable persistent CAA that I can then say, OK, go.

Trust this s.a. that would work.

Yeah, that would make it.

So that I don't have to terminate because I would I would prefer to avoid having to use ACM and terminating to a less set at application load balancer because it adds a bunch of one.

And overhead.

I would rather just wait till I set at Nginx in grass.

And then there's.

I mean, doesn't it.

Yes has all the private ECM stuff.

But I know that's very expensive.

I don't know.

And I don't know if it's in old club for what it's worth.

I'm doing something similar with the lambda right now accessing on prem where we're rolling our own s.a. and the BP BBC through like a VPN gateway site to site VPN.

And then we're Yeah, we're setting up our own CIA using Server Manager.

We're using 80, which was out of my control.

But Yeah we're using 80 to generate the certs.

OK, thanks Adam.

Yeah Now that's good.

That's good info.

Thanks any other quick questions.

All right.

Well, let's get into the first talking point, then which is going to be working from home tips.

I'd like to first open this up to Dale since you've already shared something on this that you've put together.

Let me open that up on my shared screen here.

So we can all see what that's about.

Of course.

My corporate firewall blocks Instagram.

Is this your setup down.

Yeah, it is.

That's why work from home.

So it looks like a Star Trek or something.

Pretty a pretty sweet system.

Only one monitor.

Get good scrub.

Yeah So Yeah.

Do you want to.

Do you want to narrate these slides what's going on here.

All right.

Or office actually implemented.

Work from home policy.

Due to the whole.

19 virus.

So I pretty much have started to put together just a list of thing that actually works for me in the past.

I actually had work from home prior to moving to New York for approximately eight years until a similar tip that actually worked for me.

So even when I got her one first answer did was to control the bedrooms into a little office just in case my girlfriend's here then I could actually set some boundaries as well in front of that space and make it as comfortable as possible.

So I like to be a bit more organized in what I'm doing.

I'm this freedom of words generally.

Even if you're in the office.

So I actually would start off with a natural list.

I do minty like a whiteboard to the side as well to keep a book or an iPod just to keep it kind of keep out things structured with that personal stuff.

I just use like things.

And for work related stuff I like.

That's in Europe just for everyone else's edification things Shazam app.

I do tracker.

It's not just things.

Yeah, there's an well.

Yeah thing that stuff.

So I use a lot of code based applications like everyone is more accustomed to doing as well.

So like Zoom slack jiro you know just to help with the collaboration or office metric does you use arms like generally for everything notifications just meetings as one would not just switch notice things like Blue Jeans are between blue jeans and Zoom and it suits us.

One practice.

I do maintain is getting dressed in the morning not nice.

Again Open a button down shirt but just get a little pajamas.

Take a shower.

Morning routines.

So you can mentally prepare yourself to actually get started.

Even with that.

I tend to set my desktop clear things off, make sure it's a little more organized and get seated.

You know I mentioned earlier, both setting boundaries a time again, because people think because they're working at home, you're available.

Especially if you have a family setting those boundaries making sure that your voicemail actually thinks about you're available at the times like how things sort of make people know go as far as them put it like this posted on northern and on the door with my hours of operation.

So yeah, I think that's a really good one in setting those boundaries nick and then having the conversation with your family so that they know that this is the case that things haven't really changed you just happen to be at home now.

Yeah, no.

And for my office based on this image you didn't even look at that.

I do minimum.

But at Instagram.

I put a lot of what my desktop was before that.

But I work with that system disk from Jarvis.

I switch from a dual much to a single ultra wide.

I invest a lot of time into making that space almost like a replica of what it would be like in an office space.

The music.

My laptop, whatever I would need to get things done so well.

I mean, the office or me back home.

I can still function as I would in either location.

No, I really I really like these arms for the monitors.

So you can move your screen around and get it up.

Especially when you're sitting a lot.

Having it at the right angle for your head is going to reduce some of that back pain and stress in your wrists by proper posture.

I think that's one thing that's not mentioned.

Nothing means that necessarily working from home.

But the prompt the difference between working from home often.

And the office is you have if you're not doing it often.

Yeah, pretty bad desk situation and chair situation.

So pay attention to if you start having pain in your elbows and risk because you're probably sitting in a bad post with that posture.

Yeah, I currently suffer of a slight impingement that I did therapy for and that was related to my posture at the desk.

I put Timothy Dalton like the romulo chair and I will hold my position as well just to keep some level of activity.

I like.

Yeah the I pad thing if you guys have I pad pros.

I'm not sure about the other tablets I researched it.

But I had close.

You can use that as a dual display.

I guess in 10 or 15.

It's natively supported but before that would do it.

That's awesome.

So actually my scream that I'm hearing right now is an iPad tablet.

So it's a great way to get dual displays like today if you already have.

Yeah, it's very useful, especially to see things like that while I was actually in Jamaica I had used as a second monitor as well.

That kind of simulated what I would work normally.

But my whole workflow.

The lowest I slide.

I just spoke about taking a walk was taking that break.

Step outside a lot of people don't realize that they spend so much so many hours indoors.

They don't get the sun the vital produce as much vitamin D, which may also end up with a flu season right.

Also it helps for kind of working through blockers state of mind.

Don't step outside clear your head.

Welcome back.

In and go back at it.

And then the other thing that I tend to do is to overcomplicate so we'll have chickens as well with my direct supervisor that gives the team title.

I also keep like an open zone that shows there's one guy to just jump into it.

I speak to it.

I like that.

Yeah Is that clear like I am.

Actually, that's what I'd like to talk a little bit more about I've been thinking about having as well is like for teams probably so not company wide and probably for maybe project related.

What about just having a Zoom room open that you can hang out in during the day.

You can mute yourself, you can stop the video doesn't have to be a loss of privacy or any of that.

But at least you can quickly hear any water cooler conversation that comes up related to topics on that as they want.

Now that we have that I actually implemented it.

So we use Google meat and everyone every time someone joins into two our like coffee break room as this called triggers a message in our random selection.

You can just hop in as well.

So having this as well, made it so much nicer because like in the beginning, people were just sitting there in their forest and nobody was really talking about something.

And now people joined and now we also have a calendar in mind every time that one Saudi PM and everyone is invited to join there.

So since you announce when it is.

So it's not all day.

You have it at is between specific hours kind of the day and it's open all the time.

You can point it there all the time, all day.

But like we have a dedicated session of 30 minutes where you can go in there.

That's what you announced your slack team.

Yeah you know in a general.

Yeah, we have a keyboard that are running throughout the day.

Ghost puppet because normally you in the office.

We'll just tap each other on the shore and it also helps as I'm getting lies and guess does mentally you're just not feeling alone.

Yeah, that helps.

That's a good tip.

Yeah Yeah.

And you know, this whole thing does wash your hands as good as possible.

Then if you miss it.

But I do have other tips on my Instagram that just love some of the slides about between my coupon.

I use Docker and working from more and more.

I can't stress enough that over communicate.

You That was one of my notes too.

And I think that's a really important one is that I don't think it's hard to over communicate actually and most people are actually under communicating what they're working on.

So people are not really informed on what's progressing, where they're stuck and Yeah.

Any anybody else have thoughts on that.

So one of the things my team just recently started doing.

And we really like it is there's an app on Slack called Dixie that is daily asynchronous standups and you set it up for a particular time.

And it sends each member of the team a message saying, OK, it's time for stand up you know.

And it asks you the typical three questions.

What did you do yesterday.

What are you doing today.

Do you have any blockers.

And it has.

I think it has helped a lot with getting people to write down their thoughts because when we do, we do a stand up call every day also.

But sometimes that can just be.

Oh, yeah, I was working on this other thing.

And I'm still working on it.

That's kind of it.

But getting them to write it down, gets them to go into a little bit more detail and especially with the blockers portion It is much it's much quicker to get blockers resolved when you write them down and slack and say, this is a blocker for me right now.

Someone almost always immediately goes and picks it up like if it like you know a blocker for me is I have this spread request that's waiting to be approved.

And you know nine times out of 10 somebody goes, oh, I'll go look at it.

You know cause it's right in front of them.

I'm curious about this.

And I end there.

This is probably like one of the most common app categories almost that I see for Slack.

I'm curious about anybody who's been using a tool like this for say six months or more and is still young and still sees at least let's say 80% participation in the notifications.

My inherent skepticism based on my own patterns is is like a confession here is that anything that is automated that I know is going to happen every day at the same time.

I tend to ignore as opposed to those things that are infrequent.

So this is why personally, I don't have hacks like that said a reminder every day at the same time to do something because then I just end up ignoring any anybody using this successfully in their company for a long time.

We've only been used there.

We've only been using Dixie since January, but I think as far as participation.

Our messages go out at 11:00 and we have our stand up at 11:30 and most of the stand up is going through the Dixie messages.

So if one of them isn't there you know, it's instantly you know kind of a polite name and shame kind of thing.

Why a why didn't you.

Why didn't you edit your stand.

You know.

But it's.

Oh, yeah sorry.

I forgot.

You know I got busy or whatever.

And yeah we haven't had any issues.

OK with people just forgetting about it because I mean, as long as leadership does it.

I think it tends to trickle down.

Yeah Any other.

Yeah Any other suggestions for working from home.

Brian any of your own tips or hacks you'd like to share or something in particular, you were thinking of when you asked the question in office hours General.

I actually don't have a lot of experience working in the office often.

I only work from home usually when I was sick.

So that's kind of the reason why I was asked the question.

I do like the idea of the coffee break.

I why do we already know this is like.

The office banter that we had at our office.

Yeah, so I think we're going to try today.

Think you guys a suggestion that I realized that I actually am working later into the night.

So because there's not that like a drive home thing that kind of stops you from working.

So I'm trying to figure out what I can do to fix that.

Two things.

Two suggestions on that help me at least one is making sure you set your office.

So I think developers have some different challenges from managers managers tend to live in their calendars and developers tend to just be pulled in every direction.

So it's sometimes harder to read regiment but what I was going to say is like for me on my calendar having definite work hours to find them.

So people aren't scheduling your time outside of hours.

And then the other one is disabling your slack notifications on your phone and on your desktop automatically at 5:00 PM 5:00 already or whenever it is you want your workday to stop.

Sure if you happen to be looking at it, you'll see it.

But at least hopefully it can give you the chance to close the laptop lid at a particular time and move on with your day and focus on family.

Yeah, I'd also make a comment like for the mobile apps like we use Teams internally on our organization and they usually have quiet hours.

Mm-hmm So we'll go on and/or I personally like 6:00 PM I just owe them pretty much gets news that I don't see him till the next morning, which could be a good thing or it could be a bad thing.

But it's definitely helped me when trying to disconnect.

Yeah not use like like uptime notifications and stuff.

Also anything serious like that should be set up with deletion policies actually.

Right So those should be going to page your duty or obscurity or something like that.

So that they escalate using that medium.

If it's urgent but overall, you can set, you can set different settings for different channels too.

Yeah, exactly.

Yes, we will.

We'll have a different.

Yeah, he's got a channel called alerts.

I would totally have different settings for the alerts channel than I would for the general channel or whatever.

Yeah, you can configure those settings.

And if you guys do like an uncle rotation like those of the weeks where you never have quiet hours you know.

So just think about her.

I've taken advantage of the team's feature the mobile app.

I tend to leave mine on because my team likes to just you know it even when we're outside of office hours we tend to like you know, we enjoy talking to each other.

And you know we'll put funny memes or whatever that we find.

And my software VPN particular is a night owl.

So he's up you know, every night at 10:30 doing interesting things because he is just one of those brilliant guys that is a manager.

But is smarter than I am at technical stuff.

And so he'll be up 10 30 posting links to sd 0 set.

So I like seeing that stuff.

But if it gets too much for me at any particular time, I just hit the slack is a snooze button you can say it's news on notifications for four hours or whatever.

And then that's all tend to do.

The other thing is built into OS X is the notifications menu here, you slide up and you have this.

Do not disturb.

It's also helpful.

You also can just add click it.

And then at all unreal.

So like all options for childcare.

Yeah, we'll see what other.

I jotted down some other notes.

Oh, yeah.

One thing that wasn't brought up is white boarding.

This stuff has gotten really good.

It used to be horrible.

You know you see these chicken scratches on the screen that are unintelligible.

But if you have a tablet like an and iPad Pro with an apple pencil together with either Microsoft Whiteboard, which is my personal favorite or Google jam board both of them are free.

You can do really good, high quality white boarding on these that are legible by others.

And you can then literally just if you're using Zoom, you can share share that screen on your tablet.

I would show you an example, if it's interesting.

Zoom even has fantastic white party features.

Now Yes, you does have pretty good stuff.

I would say it's a difference of if this is something you want to persist and work on or collaborate across zunes sessions, something you want to centralize like if you're using jam board.

I mean, that fits into the whole G Suite know office products.

Same with Microsoft Whiteboard.

So it's like you can continue to refer back to them and update them over a series of calls if you need to or even prepare for a call.

Yeah And as we see here.

Let's give it to you.

You just mentioned Microsoft's whiteboard and you're on a Mac.

I'm just hearing about this for the first time that I only just see developed in those 10 and I was I was curious if you use it on your Mac.

So So my point is.

So my point with this.

Why they're so usable is with a stylus.

Right So I'm using the Apple pencil on that.

And it's as good as paper for me to write on there like the quality of my I think the quality of my sketches is just as good as if I was doing it in person somewhere got it.

OK I'll just throw this in there.

I use Evernote has pretty nice.

I'll do that with the apple pencil and you can you can share those sketches.

That's true Evernote has improved their work for sketching as well.

So I haven't what.

I haven't tried to do with Evernote is collaborating on the same sketch with other people.

I don't know how that is.

I know that works well with white Ford and GM Ford.

Yeah, that's a good question because Evernote in general has been pretty poor and collaboration real time collaboration on a single note.

I always get no conflicts in that case.

I got to ask a silly question, but on the apple stylus can you.

Or maybe it's a software thing.

Can you change the shape of the tip.

And the size.

Well, yeah.

Yeah Well, that's on the software side.

So when you're using jam board or whiteboard you can change it from a pencil to a marker to highlighter to pen and different with of all those details and grids.

So it helps you draw and they also have what do you call it.

I think it was called, but they'll auto detect the shapes.

So if you draw a circle it'll make it a perfect circle.

If that's it if you like that.

Yeah, it's like snap to whatever.

Yeah Yeah.

There are other tools you can look at on the profile.

But I put itself like stability and flow.

If you're really good.

The notes is another one.

And they're all tools all there.

But make for sketching flowing from Moscow.

And does the will divert as well.

It doesn't actually have that feature.

You just measure where you can draw certainly makes a perfect circle for you as well.

Yeah What I liked about the jam board though, is like you are a sweet shop you have everything in one place.

Are you guys are you guys performing any interviews during this time.

Or are you guys going to put on all we are very firm.

We do.

I mean, we do remote interviews anyway.

So it's not really miss Messing with it.

I mean, the very final one is an in-person but we could do a remote for that to the in-person is just really do they spell.

You know do they have good hygiene.

I mean, at this point, you've talked to them a bunch of times already.

So yeah.

And you guys like the whiteboard tool.

Possibly Yeah.

We use all kinds of stuff for four interviews.

We've done some of like coding challenge type stuff that it's for some reason our legal department is it's giving us issues with that.

Yeah What the hell doesn't an announcement.

I'm actually, I recently tender my resignation at some so no one boards another company.

So I've mentioned in our whole onboarding remotely as well.

So the next three weeks or two weeks and two days I'll be there.

You mean, there is an on site.

Well, there is a revolt. You catch him.

Congrats on the change.

Thanks interesting times to start.

No Yeah but you've been remote so much.

So gear I wanted to get to your question here while we have some time.

So we you you're pretty much a regular on these office hours or haven't attended many of them.

You've heard our other talks on kind of like the Prometheus architectures.

Right And I also have to answer that a couple of times already.

But right now, I'm re implementing and rethinking.

Like I switch companies.

And we are currently like, OK.

And so that's why I like is it actually still the best thing to do.

It's just something else that I might be or should be looking out for.

So right now, my idea is like one premier just operate a protester which has a short term surge of maybe a week and then move one with long term storage, which will go entertain us, which I have never used before like I have not used it.

I use all the time Elasticsearch for long term metric data.

So yeah, I just wanted to get feedback on it and hear what you guys are doing in terms of this.

For example, which I really liked with a deadly search was that I could have all up jobs that basically would delete certain indexes after like three weeks three months for different staging clusters where the metrics are not that important for me for long term search.

But for production.

I really would like to have some metrics will like forever.

Yeah And I forget who it was there was some participant.

Now this is probably back in December, November and talked about Daniel.

So I don't have firsthand experience on Thanos.

And then there's another one competing against.

And so forget what it is.

Both of them had pros and cons and I wish I could find my notes on that plan.

Was it humor you know.

No it wasn't that one.

Anybody want to fill in while I do some rapid googling for what it's worth.

I took Erik and Andrew's advice on using it for Prometheus and that's where great for us.

I got also working with my ephemeral clusters.

So So the esfs is long live.

But the Prometheus operators are could be short lived.

So nice tool.

The interesting thing about it is it buys you a lot of runway especially since you can provision more and more IOPS as necessary and engineering time and effort is often more expensive than the provision to ops though.

So your mileage may vary in the scale of data you guys are dealing with.

I mean, if your Facebook might be different.

But for most companies.

It's not that intense.

Plus when you'd go the tiered approach the Federated approach with Prometheus and you have multiple Prometheus instances with shorter retention of Victoria metrics was the other one.

Yeah And the challenge with some of these systems is they offload the a to another system that you still now have to manage.

And my concern with having a very complex monitoring infrastructure and architecture is then staying on top monitoring your monitoring systems.

So the simpler this system is I the happier it is in my mental model right.

So for me, the long term search is more like for historical data.

And if something is basically use the class us down what happened five minutes before that like stats for what it is actually meant to be.

And for alerting all the stuff that should be in the station cluster.

So that this will stay as simple as possible.

But a long sought job search should be still there in my opinion.

Without picture metrics or something you.

So actually have found something that I will look into.

So awesome.

Thanks for that.

And yeah, I think I found the original blog post that this might have been the one that evaluate compared Thanos with Victorian metrics and the pros and cons of each and pretty like honest assessment of each one in the trade offs I am going to share that officers right now.

Thanks for that.

Yeah, I shared that as the thread of his question about I should residency.

Cool any other questions related to that or going back to the original talking point or any new questions.

It's really open ended here.

So if you haven't joined before we have quite a lot of people on the call here.

If you have any questions design decisions that you're trying to make in your organization is a great chance to get feedback on those.

And I have an abiding interest in any progress Andrews made with the get lab helm charts.

The fact that we're the ones that actually work.

It works fine.

It's just complicated.

Yeah, I was.

Yeah, I had the same experience.

OK, well complicated like like all of these different things you know like external object.

Yeah, they're like there are lots of moving parts that don't necessarily line up.

I'd love for somebody to probably have a particular, I'm not doing so well the operator I want to get the operator to work because basically, I have I no longer have access to like unlimited data about us like I used to.

So I'm running a sort of cheapo digital ocean cluster that like, well sporadically bring stuff up and down.

So I basically, I guess I just want like a scale to 0.

Get lap server and I don't have any particular like you know like it doesn't have to be any particular object storage or any particular web server I'm pretty fucked.

So if you're not.

If you're not going to use like a of USS 3 get lab a provision mineo used mineo.

Yeah, I mean, you know, it comes mineo is sort of like under the covers of a lot of like little toy projects that I end up doing.

And that's good enough.

I suppose.

And we've Yeah, we've been running mineo on our proud cluster actually.

So we SAIC has this thing called the innovation factory and part of it is the skit lab for people to use because there wasn't really a good centralized get solution that anyone could just go in and use.

But you know that spin in like beta for a year about it because we just haven't had the resources to pour into it to get it ready for any kind of a good sl low SLAs.

And so we started out just using mineo and we're still using it.

And it works fine.

It's backed by esfs.

No problems, other than the other day as in like two like Monday our esfs ran out of burst credits and everything came crashing down like to the point where it would not work at all.

So you like get lab was completely unusable.

So all I had to do was go in and up.

The And we weren't using provision day ops at all.

And so I just provisioned some my ops and it was like it was like a switch turned on.

I mean, it was like that.

Everything worked again.

You know it's.

$80 a month.

That's nothing.

You know, that's an hour and a half of my time.

You know.

So totally worth it.

I'm 100% on board with the effects.

I am not one of the doubters when it comes to, you know all kinds of people say, oh, yeah don't run your don't run your stuff on NFL don't run your database or whatever.

Yeah If you're Facebook.

Best idea.

But we've been running it on esfs.

We've been running a Boston database.

We've been running giddily which is the back service for all your know, command line for get lab.

We've been running mineo on off of VFX.

We've been running Jenkins off of VFX for over a year now.

And no problems whatsoever.

Zero zero problems.

Personally, I missed the first part they're on, where does get lab depend on something like object storage like medium well mostly for like the repositories in where it's been used a generic object storage.

It doesn't require like tell you system.

I can tell you exactly what.

Yeah, exactly.

They do elicit a dependent docs but Yeah Yeah it'll do.

Well, you have to tell it what object storage.

Oh the registry.

Sorry, that's another important part.

So artifacts backups packages planets registry and those are all those all go into buckets into three buckets.

You don't have to use those three.

You can use mineo which is this open source tool that mimics the API of S3 i.e.

That makes perfect sense when you say I was curious how they were doing get on S3 like object storage.

And it seemed like a lot of work to implement giddily itself does not which is giddily is the back service that does all the get RBC stuff.

When you say you know git clone whatever you're talking to giddily that doesn't use object storage that just uses a it's in Cuba that is it Staples that backed by a persistent volume claim and that persistent volume claim is in esfs using it has provisionally.

Any additional questions related to this or new questions.

Yeah the mineo or using exactly so mineo is a tool.

It's minack.

And you can you know, it's open source.

You can go get it on GitHub or whatever.

There's a helm chart for it and everything.

And it's a tool where you can basically host on premise.

And $3.

I they had three protocol right.

Yeah, it's exactly the same APIs Amazon S3.

So literally you can like you could have it.

You can even use something like that doesn't require local storage on the earth for it to do what it's doing obviously right.

Oh no it uses the offense to go.

I mean, it's just it all it requires is a persistent volume claim.

Right to put things in field your own history kind of.

Yeah, that's exactly what it is.

It's played out on the street.

But what's cool about it is tools that use Amazon S3 minute is a drop in replacement for them.

All you have to do is change the u.r.l. that it goes to Amazon S3 is S3 down Amazon native US or whatever.

That's Amazon.

That's the URL for us three.

If you change it toward every year you're mineo is being served at.

Everything works.

It's the it's all the same protocols it's all the same authentication.

Yeah, it works great.

My understanding of the texture of mineo isn't too radically complicated either in terms of components.

And services right.

So deploying it in Cuba and 80s with just one pot.

Yeah which is that's pretty amazing.

Yeah could he be up at all or anything like that per adding more feature functionality to this whole container base.

Sort of abstraction layer and a third, I think if I were to get more advanced on storage right now, it would be with Rooks f.

I think that's tending towards Rooks Steph is is tending to be the de facto favorite child right now for Kubernetes.

I'm actually trying to experiment with my grass spread by coastal wood and distributed optics for the law.

It seems pretty straightforward, simple enough in the face.

It looks when you're using set with that or on which one looks off.

Yeah So got like the external USB drives two works into the raspberry pies and then said that because we're called multiple types of back and providers you're sure you're thinking just stuff, though.

Yeah Yeah.

Rook rook and Rousseff are two different tools that.

And I'm not I'm not you know I know how to spell them.

That's about that's about it at this point.

But since rook is the CMC I've certified or whatever choice opera well for Kubernetes then it's going to get the best support compared to things like Gloucester which most people when I talk to them about Gloucester they say, oh, don't use that as a dumpster fire.

Yeah, it's a dumpster fire.

Totally that's the only one I can testify to firsthand experience.

I think many have actually a success story with bluster.

I'd like to hear this open b.s. stuff is like becoming popular too.

Might be worthy of keeping an eye on him.

Yeah, I think pretty well.

If you don't get a beautiful book you end up using open us.

So either one should be for any ticks.

Tips and tricks for using esfs in large capacity are like is a to have one large volume and just like segregated path based or how do you price.

Price wise, it's definitely better to have one large if s because the amount of i.e. ops you get is directly proportional to the number to the number of gigabytes of data you're storing.

So if you've got you know if you've got 10 CSS instances that all have 80 gigabytes in them the IOPS that you get from each one is tiny but if you have one CSS with 800 gigabytes in it you get a lot more.

IOPS and then, of course, you can provision more throughput till like on Monday.

I went and provisions 20 MBPS.

And I think it's probably more than I need.

But you're able to change it like once every 24 hours.

So I can bring it down.

The house like 90 bucks, man.

Yes Yeah.

So I think one would be good.

But then you have to worry about blast radius concern that's right.

If all of a sudden, a database is going crazy on your first full volume.

Other stuff doesn't work.

So I don't let anyone touch the esfs other than the offense for visionary.

No one else has access to it.

So the only things going into that you have employed using that DFS file system, you might have the noisy neighbor problem.

Yeah Yeah.

If you're sharing a big one like it when it comes to UPS like if you run out of first credits.

Definitely Yeah.

I mean, that's the problem we ran into on Monday.

As long as the first credits, then, yeah, you're fine.

That's what we.

That was one of the things that we ran into with Prometheus in the early days was that the volume was so small that we didn't have a diverse credits.

So we ended up having to artificially we will in our case, we provision more IOPS.

I know Andrew, you've also said, you can just write a 0 0 seal file just garbage data to increase the file size of the Earth to increase the size of the file system.

Yeah, you can do that to you like you've got to just work out the pricing would you expect me to resist paying for it and keep the architecture.

Simple then.

Yeah imaging.

Yeah And I know I didn't do that when it came to actually needing to fix this thing when it all came crashing down on Monday.

I just provisioned for throughput.

Yeah, it's kind of like a provision I have on already.

Yes It's just more expensive than just expanding the disk size.

Mm-hmm So yeah I've actually been considering and it's going to be a battle getting my team on board.

Is there you know they think our audience is God's greatest gift to humanity.

But with the way, I'm not I don't see the industry.

But with the way my industry and the government's face is going they really put a premium on making things.

Cloud agnostic.

So I've actually been doing been a lot more interested and doing some more work on looking at everybody tells me not to run a database in communities.

But I kind of want to run a database and companies.

So you should look at post stock SQL like you never lived until you like upgraded your whole Cocker cluster in like 10 minutes worth of planning and five minutes of execution.

And like Harvey any like even a blip of an outage.

So it does.

Man it feels pretty good.

Thompson So well that Wall Street felt like something else after they have shots.

Borkowski I got released.

I mentioned right.

Yeah, I had happily customized charts.

Yeah well this has got all kinds of databases.

Yeah So this is open.

I've heard of this before.

Yeah And its operators for communities to manage these business logic for managing these services on Cuba.

Oh, that's awesome.

Well, you know, I can't speak from firsthand account.

I just know that that's what their prerogative.

She just my business model.

I bought that site.

And I'm just sort of unclear where they're coming from.

But they make some really great stuff.

To pod disruption budgets help.

I mean, can you set them up to like cap the amount of data storage access.

No, that's more about how frequently Kubernetes can nuke it and move it somewhere else.

Oh, OK.

Yeah Yeah.

So it's maintaining stability of the service.

Yeah over rebalancing pods in the cluster that's beautiful.

I am definitely to check this out.

Well cool.

Yeah, I mean, we don't run production grade databases in the cluster but we've had no problem with staging and other stuff in the cluster.

So why mostly personnel issues like we don't have enough time to really understand that system.

But our address is well understood.

So more of just a it's a safety fallback for us, unless we can actually engineer that test it, build it, make sure it works well and actually monitor it and operate it.

Well, it's a little more nuanced worth.

It's not worth the risk to us to not use it.

Yes that's fair.

If you've got if you've got a team like 10 guys you can you can fit that in.

Go for it.

We've got three guys.

That's not enough.

That's a really good point.

And it also goes back to that wise comment Chris fouls that thinks that you know if you're introducing software like this and you don't have the resources to manage the lifecycle of it.

It's going to be in the critical path and the problem.

That's my paraphrasing his statement, which was, if you can't stand the heat, get out of the kitchen.

Yeah, more or less awesome, guys.

So that brings us to the end of the hour.

Thank you for sharing all the tips from working from home.

Brian, I expect to you to be productive.

Now during the next two weeks as a result of this.

Thanks, everyone, for sharing.

Remember to register for our weekly office hours if you haven't already.

Go to cloud plus slash office hours a recording of this call will be posted to the office hours channel as well as syndicated to our podcast at podcast.asco.org dot cloud posse so you can subscribe using whatever podcast software you use.

See you next week.

Same place, same time.

All right, guys.

But I use.