Effortless Helm Chart Deployments

adminCI/CD, DevOps, Meetup0 Comments

Learn how to deploy complex service-oriented architectures easily using Helmfiles. Forget umbrella charts and manual helm deployments. Helmfile is the missing piece of the puzzle. Helmfiles are the declarative way to deploy Helm charts in a 12-factor compatible way. They're great for deploying all your kubernetes services and even for Codefresh continuous delivery to Kubernetes. We'll show you exactly how we do it with a live demo, including public repos for all our helmfiles.

Top 5 DevOps Bad Habbits

adminDevOps, OpEd0 Comments

Recently, we were asked to name our top (5) DevOps “Worst Practices” (or anti-patterns). Here's what we came up with…

#1. Not looking outside the organization to see how others are solving the problem. Always building new things rather than looking for readymade solutions (open source, SaaS, or enterprise offerings) which leads to piles of technical debt & inevitable snowflake infrastructures.

#2. Not building easy tools that the rest of the company can use. We must never forget who we are serving. Developers are our customers too.

#3. Not treating DevOps as a shared responsibility. It needs to be embedded into the engineering organization, not relegated to a select few individuals. “DevOps” is more of a philosophy than a job title.

#4. Not treating Infrastructure as Code. We call this new paradigm GitOps, where Git system of record for all infrastructure and CI/CD is our delivery mechanism.

#5. Never ever commit to `master`. We don't do it in regular software projects, we shouldn't do it for ops. Everyone should be following the standard Git Workflow on their Infrastructure Code (Feature Branching, Pull Requests, Code Reviews, CI/CD). This increases transparency and helps the rest of the team stay up to date with everything going on.

Unlimited Staging Environments

adminCI/CD, DevOps, Slides0 Comments

How to run complete, disposable apps on Kubernetes for Staging and Development

What if you could rapidly spin up new environments in a matter of minutes entirely from scratch, triggered simply by the push of a button or automatically for every Pull Request or Branch. Would that be cool?

That’s what we thought too! Companies running complex microservices architectures need a better way to do QA, prototype new features & discuss changes. We want to show that there’s a simpler way to collaborate and it’s available today if you’re running Kubernetes.

Tune in to learn how you can assemble 100% Open Source components with a CodeFresh CI/CD Pipeline to deploy your full stack for any branch and expose it on a unique URL that you can share. Not only that, we ensure that it’s fully integrated with CI/CD so console expertise is not required to push updates. Empower designers and front-end developers to push code freely. Hand it over to your sales team so they can demo upcoming features for customers! The possibilities are unlimited. =)


Feature Branching & Pull Requests (Git Workflow)

adminDevOps0 Comments

Feature Branches & Pull Requests


When you're working on a project, there will be a bunch of different features or ideas in progress at any given time, not all of which are ready for prime time. Further more, as business priorities change, you might need to prioritize certain features and put others on the back burner. 

At the same time, business requirements mandate that you have a stable version that can be deployed at any given time. We know code can never be entirely bug free. Further more, once deployed there can be unintended consequences. Other times, managers simply change their mind and decide that a certain feature was premature or unnecessary. To mitigate the impact of these events, we need the ability to rollback a feature or cut-bait.

TL;DR: If everyone is working on the same branch such as master, it pollutes the commit history making it all but impossible to figure out which commits relate to specific features making rollbacks impossible. 


To solve this problem, the standard workflow called branching should be used religiously. Any time a new feature is developed it must be worked on in a separate branch. When you create a branch, you're creating an environment where you can freely test out new ideas without impacting others because changes made on a branch don't affect the master branch (or any other one). 

Further more, no two developers should ever commit to, or work on the same branch at the same time (unless they have permission with the branch stakeholder). Instead, they should create a new branch. 

The next thing that needs to happen is that the master branch is treated as the Holy Grail. Every effort is made to ensure it's stable and can be deployed to production at any time.

Once a feature is considered ready, the developer submits a Pull Request (or PR) and assigns it to a Subject Matter Expert (SME) or peer for review.

On the surface, this is what a well-formatted Pull Request looks like:

A Pull Request allows many things to happen:

  • Title: A “human readable” title that represents the feature

  • Description: A long description that details What was changed, Why it was deemed necessary, Who should review it, and any other References that might be useful (E.g. Jira ticket)
  • Comments: let anyone provide arbitrary feedback viewable by everyone.  
  • Diffs: show what changed between this feature and the current master branch
  • Formal Code Review Process:  let multiple people contribute to the code review process by submitting comments on a line-by-line basis. Having these code reviews formally documented serves as an excellent teaching tool. Over time, the reviews become faster and faster as developers learn what is expected.

  • Merging: Once the PR is approved, the developer can squash and merge their code into the master branch. Squashing allows the master branch to have a very clean commit history where every commit corresponds to a PR. 

  • Clean Commit History: means that every change to the master branch is documented and justified. No one is sneaking in changes.

  • History of Features and when they were added

  • Reverting: If a feature needs to be removed, with the click of a single button it can be removed from the master branch

Technical Details

Create a Branch

Whenever you begin work on a new feature or bugfix, it's important that you create a new branch. Not only is it proper git workflow, but it also keeps your changes organized and separated from the master branch so that you can easily submit and manage multiple pull requests for every task you complete.

To create a new branch and start working on it:

# Checkout the master branch - you want your new branch to come from master
git checkout master

# Pull down the latest changes
git pull origin master

# Create a new branch, with a descriptive name  (e.g. implement-xyz-widget)
git checkout -b newfeature

Now, go to town hacking away. When you're ready, push the changes up to the origin.

git push origin newfeature

Submitting a Pull Request

Prior to submitting your pull request, you might want to do a few things to clean up your branch and make it as simple as possible for the original repo's maintainer to test, accept, and merge your work.

If any commits have been made to the upstream master branch, you should rebase your development branch so that merging it will be a simple fast-forward that won't require any conflict resolution work.

# Fetch upstream master and merge with your repo's master branch
git pull origin master --rebase

Follow the prompts to correct any code conflicts. Any file that is conflicted needs to be manually reviewed. After you fix the problems run:

 git add filename
git rebase --continue

Once that is happy, push the rebased changes back to the origin.

git push origin newfeature -f

Then follow these instructions once you're ready:


Pull Request Template

Use the following markdown template to describe the Pull Request.

## what
* ...high-level explanation of what this PR accomplishes...

## why
* ...business justifications for making the changes...

## references
* ...related pull requests, issues, documents, or research...

Prepared By

Erik Osterman
Cloud Posse, LLC

310 496 6556

312 Arizona Ave
Santa Monica, CA

How to Easily Pair Program on the Console

adminDevOps0 Comments

Peer Console Development (tmate)

Tmate is a way to share a console window or set of console windows. It’s basically a fork of tmux that makes sessions network enabled. It doesn’t matter where the participants are located - inbound firewall rules don’t interfere since it’s all outbound connections. 

Install tmate

brew tap nviennot/tmate
brew install tmate

Start the SSH Server

tmate -S /tmp/tmate.sock new-session -d

Wait for a connection

tmate -S /tmp/tmate.sock wait tmate-ready 

Get the SSH Command

tmate -S /tmp/tmate.sock display -p '#{tmate_ssh}'

Share the output of the above command with someone you want to share the terminal window with. You should both run the command. The person who runs the SSH Server is the host of the session.

# Example session: (now expired)
ssh q9YsisZyUNtI3rvJcDbG1pAHu@sf1.tmate.io

Interacting with Tmate

^b + c create a new window
^b + n move to next window
^b + p move to previous window

The “Holy Grail” of DevOps

Erik OstermanDevOps2 Comments

Learn about some of the “best practices” for running a modern engineering organization that we practice at Cloud Posse. “Best Practices” are opinionated & proven strategies/tactics used to achieve some desired outcome.
These slides are a small part of what we think it takes to embrace a DevOps culture. It starts with the “Organization” embracing change and laying a foundation that will support the cross-disciplinary practice of DevOps.

Building a DevOps Culture

Erik OstermanDevOps0 Comments

A question that often comes from well-established organizations with “mature” infrastructures is the following:

How can an organization instill a new engineering culture where developers and operations are working with each other and not against each other?

We affectionally call this “DevOps” movement; it's a culture where developers and ops work together and not against each other. Often their distinct roles are indistinguishable. Developers are confident on the command-line just as much as ops are confident in the IDE.

The key to succeeding with Devops is demonstrating to developers that it will actually make their job easier, such as when they can debug issues. Likewise, Ops needs to see the developers as a resource who can reduce the number of sleepless nights they experience as a result of failed deployments and buggy code. The role of ops in a DevOps culture is to enable devs to operate more efficiently. The role of devs in the organization is to build applications which are easily deployable.

Here’s what needs to happen. Ops needs to take a first step in standardizing the way software gets deployed. Start with taking a look at the current open source tools available. Choose one. Then they take those recipes and build a local development environment using something like Docker Compose or Vagrant so that developers can start getting familiar with it. Next, developers need to embrace the local dev environment over “native” environments (e.g. those that take a day of configuration and downloading packages). Through this process, they build up operational competency in debugging issues in environments that mirror production. After several months of operating like this, developers should shadow ops in certain roles, such as deploying software.

Once the above system is in place, the next step is to increase monitoring coverage and make this transparent to everyone in the org. In addition to your standard checks (like Nagios), you’re going to need to deploy something like DataDog or NewRelic. This gives your developers insights into how their apps are function in production and the necessary information to diagnose bugs. Tasks get created for every warning or error in production that is not handled in the proper manner. Once these alerts are calibrated to less than a few critical incidents per week, they should get wired up with PagerDuty. This is where the most resistance is usually encountered, but this is what keeps everything going. Without skin in the game, developers have no incentive make sure their code is highly reliable and ops are tied in what they can do to fix problems. But with only a few critical incidents, devs will be quickly motivated to fix the bugs and silence the alarms.

The last triumph in this conversion is to achieve “Continuous Integration” (CI) coupled with “Continuous Deployment”. Before this can even be remotely considered, considerable unit test coverage is needed. Stay tuned for more posts on these topics.