November 2018 Edition of the SweetOps Newsletter

adminNewsletters0 Comments



Welcome to the November 2018 edition of the SweetOps newsletter.

We've got a lot to share with you this month as we've been exceptionally busy! Overall, our emphasis this month has been on growing the community around our open source adoption and investing heavily in Atlantis for terraform GitOps-style automation.



New Terraform Modules

  1. terraform-aws-iam-account-settings – Terraform module to provision general IAM account settings. It will create the IAM account alias for pretty login URLs and set the account password policy.
  2. terraform-aws-iam-user – Terraform Module to provision a basic IAM user suitable for humans. Supports automatic password encryption using Keybasepublic keys.
  3. terraform-null-smtp-mail – Terraform module to send transactional emails via an SMTP relay directly from within terraform. This is perfect for sending teams email notifications when infrastructure changes.


You can find all of the cloudposse terraform modules in the official terraform module registry.

Important Terraform Module Updates

  1. terraform-aws-rds-cluster – we've added support for Aurora Serverless as well as fixed some idempotency issues raised by the community.
  2. terraform-aws-cloudfront-s3-cdn – we've fixed support for regional S3 endpoints with the CloudFront CDN
  3. terraform-aws-elastic-beanstalk-environment – we've received a handful of PRs to extend the module capabilities with better support for autoscaling, nodejs, ELB security groups, and cloudwatch logs.
  4. terraform-aws-dynamodb – we've added support for local secondary indexes

Many thanks to @johncblandii, @br0nhy, @bober2000, @pabardina, @rverma-nikiai, @aknysh, @jamisonhyatt and the many others who took the time to open issues, submit pull requests and review code.


GitOps with Atlantis and Terraform


We recently held a meetup during #connectweek in Pasadena (CA) where we gave a live demo using Atlantis with Terraform to provision AWS user accounts using only Pull Requests. Atlantis is the secret for enabling teams to collaborate on Terraform. We have our own fork of atlantis that we're maintaining until the essential security features we introduced get upstreamed.


Slides from the presentation are available on our blog.


Interesting Links & Projects

Here are some interesting links that circulated this month in our slack team.

  1. Very cool Open Source SIEM security and compliance dashboard for surfacing events.
  2. Interesting workaround to achieve Tillerless Helm today by running tiller on localhost.
  3. HashiCorp, the company behind Terraform, has raised an additional $100M
  4. 10 Habits DevOps practitioners must break
  5. Monitoring dashboard for Kubernetes Jobs that makes it easy to see which are running and if their latest status was “succeeded” or “failed”.
  6. Two Objects not Namespaced by the Linux Kernel and therefore shared by linux containers.
  7. Malicious Life Podcast – The True stories behind the world of cybercrime

For more links, subscribe to our @cloudposse twitter feed.

Follow us on Twitter @cloudposse

New Alpine Package Repository


As part of Geodesic, our framework for provisioning and automating infrastructure, we needed a better way to distribute our toolchain (aws-vault,   chamber, kops, gomplate, etc) that was fast and reliable that made it easy to version pin tools for individual customers. Previously, we shipped all tools with the geodesic docker base image pinned to a specific release, but this was bad because it forced a customer to upgrade their entire toolchain when they just wanted one upgraded package.


The fix was to distribute all of our tools as versioned alpine packages. Our alpine repository is available at More instructions for configuring this repository can be found in the github repository where we define all packages.


What does this mean if you use the make based package installer? Not much. We'll continue to support Makefile based package installations because we need a way to install these packages natively on OSX/Linux/Windows/etc that is not alpine specific. In the future, we might distribute the packages for Debian and RPM-based distros as well.


Terragrunt with Geodesic


Terragrunt is a popular tool for automating terraform deployments. It makes it easy to reuse modules, initialize terraform state, and orchestrate complex multi-stage terraform build-outs.


For a demo we recently did on using Atlantis with Terraform, we put together an example of using Terragrunt on Geodesic. Long and short of it is that it's easy, and there's not much to it. For an example, refer to our reference architectures. Here's an example of adding a user using terraform and terragrunt in combination with our terraform-root-modules.


Slack Community

Over the past month, we've seen a sharp uptick in the number of daily conversations especially in the #terraform, #release-engineering, and #kubernetes channels. We're grateful for everyone helping out to answer questions and sharing interesting articles. If you haven't already, sign up for the SweetOps slack team today.

Join Slack Community

Reviews & Testimonials

We've set up a place for our community members to leave us testimonials. If our modules or slack community have helped you, we would love to know. A few words from you would totally make our day and help us reach others.

Please leave us a testimonial, if you've found some of our open source repos helpful.


All the best,


Erik Osterman

CEO / Founder, Cloud Posse, LLC



How to use Terraform with Teams using Atlantis (#GitOps)

adminMeetup0 Comments

GitOps is where everything, including infrastructure, is maintained in Git and controlled via a combination of Pull Requests and CI/CD pipelines. Reduce the learning curve for new devs by providing a familiar, repeatable process. Use Code Reviews to catch bugs and increase operational competency. Provide transparency to the rest of the team with Pull Requests. This presentation had a live demo of using Atlantis with Terraform that showed how to easily add and remove users from AWS IAM safely & securely using simply GitHub Pull Requests.

Erik Osterman is the founder of Cloud Posse, a DevOps professional services company that specializes in cloud migrations and release engineering. Previously he was the Director of Cloud Architecture, for CBS Interactive where he led cloud strategy across the organization.


Effortless Blue/Green Deployments on Kubernetes with Helm

adminMeetup0 Comments

Last night was our first ever Pasadena “DevOps Mastermind” meetup.

First speaker up was Dan Garfield. He talked about how to achieve Blue/Green deployments. Blue/Green has been around for a long time but what are the “best practices” when using Kubernetes? How does it change when using Helm? Last night we learned from Dan the differences as he demonstrated how to pull it off effectively with repeatability using Codefresh. When using Helm, the picture changes slightly, keeping a history so rollbacks work properly is critical and requires structuring your Helm Chart accordingly. Check out the slides!

Dan Garfield is a Google Developer Expert, Chief Evangelist of Codefresh, and Kubernetes, Helm, Istio, and Docker meetup organizer. His talks have been featured at Kubecon, Swampup, DeveloperWeek, and many other places. He focuses on DevOps, and Deployment Strategies in a micro-service world.

Effortless Helm Chart Deployments

adminCI/CD, DevOps, Meetup0 Comments

Learn how to deploy complex service-oriented architectures easily using Helmfiles. Forget umbrella charts and manual helm deployments. Helmfile is the missing piece of the puzzle. Helmfiles are the declarative way to deploy Helm charts in a 12-factor compatible way. They're great for deploying all your kubernetes services and even for Codefresh continuous delivery to Kubernetes. We'll show you exactly how we do it with a live demo, including public repos for all our helmfiles.

SweetOps Town Hall Meeting

adminMeetup0 Comments

Today we had our first “Town Hall” meeting where members of our SweetOps community ( got together on a Zoom conference call to talk shop.

Remember to vote when we should have our next call.

Discussion Points

  • GitOps – CI/CD Automation of Terraform
  • Git ChatOps
  • OAuth2 Proxy
  • Govcloud

Pain Points

There were a few shared pain points by members.

  • Keeping modules up to date is tedious, especially in busy environments where modules are released frequently. We wish there was a way to identify out of date modules and easily update those module references. Erik mentioned that he submitted a “Feature Request” for Dependabot to support HCL. Please 👍 if you think this would be awesome!
  • Loren mentioned that using Open Source in Govcloud is difficult because Govcloud lags way behind the commercial AWS cloud in features. For example, EC2 instance tags cannot be set on launch. Other times, API endpoint URLs just don't work because they've been hardcoded to the ones used by the commercial cloud offering.
  • Erik mentioned that OAuth2 Proxy appears to be an abandoned project. With over 90+ open pull requests, broken OpenID Connect support and no new releases in over a year, maybe it's time for a new maintainer. Apparently, others have suggested this too.


A number of technologies came up in our discussions


Other Links



  • Hold these “Town Hall” meetings more often (perhaps weekly)  and at different times so that we can include members in all geographies and time zones.
  • Ensure we record and publish the next call

Geodesic Logo!

adminUncategorized0 Comments

We've finally settled on a logo for Geodesic, our solution for implementing rock-solid cloud architectures.

In mathematics, a geodesic line is the shortest distance between two points on a sphere. It's also a solid structure composed of geometric shapes such as hexagons.


We like to think of geodesic as the shortest path to a rock-solid cloud infrastructure. The geodesic logo is a hexagon with a cube suspended at its center. The cube represents this geodesic container, which is central to everything and at the same time is what ties everything together.


But look a little closer and you’ll notice there’s much more to it. It's also an isometric shape of a cube with a missing piece. This represents its pluggable design, which lets anyone extend it to suit their vision.

Top 5 DevOps Bad Habbits

adminDevOps, OpEd0 Comments

Recently, we were asked to name our top (5) DevOps “Worst Practices” (or anti-patterns). Here's what we came up with…

#1. Not looking outside the organization to see how others are solving the problem. Always building new things rather than looking for readymade solutions (open source, SaaS, or enterprise offerings) which leads to piles of technical debt & inevitable snowflake infrastructures.

#2. Not building easy tools that the rest of the company can use. We must never forget who we are serving. Developers are our customers too.

#3. Not treating DevOps as a shared responsibility. It needs to be embedded into the engineering organization, not relegated to a select few individuals. “DevOps” is more of a philosophy than a job title.

#4. Not treating Infrastructure as Code. We call this new paradigm GitOps, where Git system of record for all infrastructure and CI/CD is our delivery mechanism.

#5. Never ever commit to `master`. We don't do it in regular software projects, we shouldn't do it for ops. Everyone should be following the standard Git Workflow on their Infrastructure Code (Feature Branching, Pull Requests, Code Reviews, CI/CD). This increases transparency and helps the rest of the team stay up to date with everything going on.

Unlimited Staging Environments

adminCI/CD, DevOps, Slides0 Comments

How to run complete, disposable apps on Kubernetes for Staging and Development

What if you could rapidly spin up new environments in a matter of minutes entirely from scratch, triggered simply by the push of a button or automatically for every Pull Request or Branch. Would that be cool?

That’s what we thought too! Companies running complex microservices architectures need a better way to do QA, prototype new features & discuss changes. We want to show that there’s a simpler way to collaborate and it’s available today if you’re running Kubernetes.

Tune in to learn how you can assemble 100% Open Source components with a CodeFresh CI/CD Pipeline to deploy your full stack for any branch and expose it on a unique URL that you can share. Not only that, we ensure that it’s fully integrated with CI/CD so console expertise is not required to push updates. Empower designers and front-end developers to push code freely. Hand it over to your sales team so they can demo upcoming features for customers! The possibilities are unlimited. =)


Quip Plugin for WordPress

adminProjects0 Comments

We just published our New WordPress Plugin for @quip. Easily embed quip documents in any #wordpress post or page. Check it out here!

WordPress Plugin Directory:


Screenshot 1: Example Usage

Screenshot 2: Admin UI


To embed the content of a Quip document into a WordPress page or blog post, install the plugin and then use the quip shortcode.

quip shortcode accepts two attributes and has the following format:

[quip id="mWnnAszre3MW" ttl=7200] 


  • id (Required) – The ID of the Quip document (e.g.
  • ttl (Optional) – Time-To-Live in seconds. After the first request to the Quip API, the plugin caches the content of the document (HTML and images) for the specified amount of time (seconds). All consecutive requests to the same page or blog post will not call the Quip API again but instead will retrieve the document from the internal cache, making the pages faster. After the ttl expires, the plugin will call the Quip API and cache the result again. If the ttl attribute is not provided, the default value of 7200 seconds (2 hours) is used. You can change the default value in Quip Settings (menu Settings/WP Quip). If ttl is set to 0, the plugin will not cache the Quip API responses, and every request to the WordPress page or blog post will call the Quip API.



Feature Branching & Pull Requests (Git Workflow)

adminDevOps0 Comments

Feature Branches & Pull Requests


When you're working on a project, there will be a bunch of different features or ideas in progress at any given time, not all of which are ready for prime time. Further more, as business priorities change, you might need to prioritize certain features and put others on the back burner. 

At the same time, business requirements mandate that you have a stable version that can be deployed at any given time. We know code can never be entirely bug free. Further more, once deployed there can be unintended consequences. Other times, managers simply change their mind and decide that a certain feature was premature or unnecessary. To mitigate the impact of these events, we need the ability to rollback a feature or cut-bait.

TL;DR: If everyone is working on the same branch such as master, it pollutes the commit history making it all but impossible to figure out which commits relate to specific features making rollbacks impossible. 


To solve this problem, the standard workflow called branching should be used religiously. Any time a new feature is developed it must be worked on in a separate branch. When you create a branch, you're creating an environment where you can freely test out new ideas without impacting others because changes made on a branch don't affect the master branch (or any other one). 

Further more, no two developers should ever commit to, or work on the same branch at the same time (unless they have permission with the branch stakeholder). Instead, they should create a new branch. 

The next thing that needs to happen is that the master branch is treated as the Holy Grail. Every effort is made to ensure it's stable and can be deployed to production at any time.

Once a feature is considered ready, the developer submits a Pull Request (or PR) and assigns it to a Subject Matter Expert (SME) or peer for review.

On the surface, this is what a well-formatted Pull Request looks like:

A Pull Request allows many things to happen:

  • Title: A “human readable” title that represents the feature

  • Description: A long description that details What was changed, Why it was deemed necessary, Who should review it, and any other References that might be useful (E.g. Jira ticket)
  • Comments: let anyone provide arbitrary feedback viewable by everyone.  
  • Diffs: show what changed between this feature and the current master branch
  • Formal Code Review Process:  let multiple people contribute to the code review process by submitting comments on a line-by-line basis. Having these code reviews formally documented serves as an excellent teaching tool. Over time, the reviews become faster and faster as developers learn what is expected.

  • Merging: Once the PR is approved, the developer can squash and merge their code into the master branch. Squashing allows the master branch to have a very clean commit history where every commit corresponds to a PR. 

  • Clean Commit History: means that every change to the master branch is documented and justified. No one is sneaking in changes.

  • History of Features and when they were added

  • Reverting: If a feature needs to be removed, with the click of a single button it can be removed from the master branch

Technical Details

Create a Branch

Whenever you begin work on a new feature or bugfix, it's important that you create a new branch. Not only is it proper git workflow, but it also keeps your changes organized and separated from the master branch so that you can easily submit and manage multiple pull requests for every task you complete.

To create a new branch and start working on it:

# Checkout the master branch - you want your new branch to come from master
git checkout master

# Pull down the latest changes
git pull origin master

# Create a new branch, with a descriptive name  (e.g. implement-xyz-widget)
git checkout -b newfeature

Now, go to town hacking away. When you're ready, push the changes up to the origin.

git push origin newfeature

Submitting a Pull Request

Prior to submitting your pull request, you might want to do a few things to clean up your branch and make it as simple as possible for the original repo's maintainer to test, accept, and merge your work.

If any commits have been made to the upstream master branch, you should rebase your development branch so that merging it will be a simple fast-forward that won't require any conflict resolution work.

# Fetch upstream master and merge with your repo's master branch
git pull origin master --rebase

Follow the prompts to correct any code conflicts. Any file that is conflicted needs to be manually reviewed. After you fix the problems run:

 git add filename
git rebase --continue

Once that is happy, push the rebased changes back to the origin.

git push origin newfeature -f

Then follow these instructions once you're ready:

Pull Request Template

Use the following markdown template to describe the Pull Request.

## what
* ...high-level explanation of what this PR accomplishes...

## why
* justifications for making the changes...

## references
* ...related pull requests, issues, documents, or research...

Prepared By

Erik Osterman
Cloud Posse, LLC
310 496 6556

312 Arizona Ave
Santa Monica, CA