Enterprise DevOps Accelerator

If you are an enterprise, you need to leverage AWS Organizations, Organizational Units (OUs), and member accounts to segment workloads. This is standard in every one of our engagements as part of our Reference Architecture.

Operating Multiple Organizations

It's not uncommon for enterprises to manage more than one top-level AWS root account.

If you currently operate multiple organizations, we can help! We have implemented multiple patterns over the years. Here are some examples:

Due to acquisitions

Enterprises frequently acquire other companies operating in AWS. Simply adopting the member accounts of the acquired company can be a disruptive operation. Furthermore, acquired companies sometimes operate independently for some time after acquisition.

Due to security segmentation

We've seen some organizations use separate organizations for the SecOps, ensuring that they are fully autonomous.

Due to stage segmentation

We've helped customers who use a dedicated AWS organization per stage (dev, staging, production). This is ideal from an isolation perspective. It means that changes made in the root “dev” organization cannot affect anything production-facing.

For companies managing a huge volume of accounts, Terraform is an excellent choice for managing it. We manage the entire lifecycle of the AWS accounts with terraform (not ControlTower or AFT) in our Reference Architecture.

Please note that AWS ControlTower does not support terraform (as of 2022). The recent release of Account Factories for Terraform (AFT) ^[1]https://learn.hashicorp.com/tutorials/terraform/aws-control-tower-aft is actually not so much about creating accounts with terraform as it is about enabling ControlTower to establish a terraform-baseline (E.g. a state backend) within the member account. Most of AFT can be provisioned with terraform, but just bear in mind, the product of the module is not an account, but a factory for creating accounts by committing files to a VCS system.

To scale the adoption and maintenance of your AWS organization with Infrastructure as Code, the business needs to adopt the pattern of building a reusable catalog of services, and landing zones that provision a baseline of services and guardrails so that teams can operate autonomously following established best practices.

A well-designed service catalog can be centrally managed, and changes can be pushed out to consumers of the service catalog in an automated fashion.

Our Reference Architecture implements the most comprehensive Service Catalog for services on AWS using terraform. Not only that, all of our services are Open Source and licensed under APACHE2.

We work with many companies operating in regulated industries like healthcare or e-commerce, or with public companies that need to maintain their SOC2 Type II compliance while modernizing how they approach cloud infrastructure on AWS.

Our Reference Architecture supports the full suite of AWS security-oriented products (e.g., Audit Manager, SecurityHub, GuardDuty, Inspector, Macie, et al) and is implemented as part of our Foundational Security & Compliance pillar.

Successful enterprises frequently have workloads running in less optimal configurations managed on systems such as dedicated EC2 instances and using classic configuration management tools (e.g., Chef, Puppet, Ansible, cfEngine, Salt, etc) to manage them.

To modernize this, we recommend moving these workloads into containers, running them on platforms like EKS (Kubernetes) or ECS, and deploying them with standardized methods shared by all services.

We support this in our Reference Architecture as part of our Foundational Platform and Foundational Release Engineering pillars.

Building this is the easy (fun) part. Moving workloads is the hard part. You need a plan for how to move workloads from legacy environments into newer ones. The problem is only magnified when you have petabytes of data stored across possibly hundreds of S3 buckets, databases, etc. A proper plan will take into account many factors. The more complex the architecture, the more traffic that is served, the more data stored, the more regulated the industry, and the less tolerance for downtime — the more preparation is needed for successful execution.

When you work with us as part of our Enterprise Accelerator, we work with you to construct a migration plan broken down into stages. Here's a simple checklist that shows an example of a high-level plan for more straightforward migrations.

Modern cloud infrastructure never depends on just one provider like AWS. It will integrate services like Datadog, OpsGenie, Okta, GitHub, etc.

As a result of all the work we've performed for Startups and Enterprises, we support dozens of integrations managed by Terraform as part of our Reference Architecture for terraform.

 

 

We've seen many of our Enterprise-focused SaaS customers have the need to deploy dedicated single-tenant AWS accounts (e.g. dedicated AWS accounts for customer subscriptions). This is usually to satisfy some regulatory requirements of their customers so that data is never collocated with other tenants. Other times, it's to control the cadence of software releases. For example, IoT or medical device companies may need to carefully control the schedule when production environments are updated, and those changes could happen out of phase with other customers.

We readily support this pattern as part of our Reference Architecture with terraform.

Startups are frequently put under the microscope when they are going through the due diligence process as part of a potential acquisition or funding round. The infrastructure needs to be sound.

Companies leveraging our Reference Architecture for terraform will pass these audits with flying colors. Honestly, they may even ask you for a referral. 😉

If you need to get something up fast (e.g., within 2-4) weeks, our Bootcamp or Jumpstart accelerators are ideal, depending on your level of experience and budget. These solutions leverage our proven reference architecture for terraform, but to deliver them quickly are delivered prebaked with all of our standard recommendations, and you can customize anything before going into production, or leverage our professional services as needed to make modifications for you.

Hiring for DevOps is probably the hardest, most competitive position for companies. The best candidates receive multiple offers, with steep price tags. Recruiters charge a 20-30% finders fee on top of a $150K-$250K base salary for a US-based Senior DevOps Engineer and don't guarantee an outcome on that investment. On top of that, the reality is that not all businesses have problems attracting talented engineers, and even if you do, they may churn after only a year.

Why compromise?

When you work with Cloud Posse, we provide a guaranteed outcome on time and within budget. We have some of the best engineers working on these problems, and we're always there for you.

The unfortunate consequence of successful businesses that move quickly is that they pile up technical debt in the form of outdated technology choices, software that is many versions behind the latest releases, known/unknown security vulnerabilities, and Rube Goldberg apparatuses that only the inventors could understand. As technology teams churn, newer generations of engineers are left to care for this baggage. On the surface, business operations may seem fine, but there's a dumpster fire simmering. The team's time will be spent unnecessarily putting out fires as they come up, unable to focus their attention on innovation and solving the business's core needs.

When you work with Cloud Posse, we can help you plot a course to eradicate the technical debt associated with your cloud infrastructure deployments. We start with a net new AWS organization and migrate over only services that you depend on. What's left is an old tainted AWS organization that can be sunset and a new pristine organization enabling developers to unencumbered to deliver software to customers faster and more reliably.

By in large most companies build their infrastructure using entirely in-house resources. They either pull from their existing talent pool or hire a team to build it. The advantage is that the business may think it knows exactly what it needs, so it starts building it immediately. It's a rewarding process for engineers because developers are inventive, natural creators who love to build new things.

To go this route, the business must allocate a budget, pull the resources aside to focus entirely on building out the next generation of its platform, and then construct the plan for getting there.

The risk is that the company hasn't done this before. While the team is intelligent and accomplished, they lack the end-to-end plan for getting there and likely underestimate the time & effort required to complete the project. In engineering, it's too common for engineers to underestimate the level of effort and the impact of distractions on timelines and overestimate the likelihood of succeeding the first time.

Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.^[2]

Douglas Hofstadter 

More commonly, we see that the engineers get pulled back into firefighting mode. Now the business has two problems. They are maintaining the old infrastructure while attempting to innovate on the new infrastructure, but the team is exhausted from fighting fires.

Once you've built this shiny new infrastructure, it moves into the second stage: maintenance. This is a boring phase compared to the stage of building it. We see many companies hemorrhage engineers, once the infrastructure is built because the fun is over; the company is left holding the bag, and the persons who built the infrastructure no longer stick around to maintain it.

On the other hand, working with a DevOps Accelerator means you have the plan already in place, and the plan has been vetted and iterated on with each implementation executed by the DevOps Accelerator. This continual learning leads to a better outcome with less risk. The time estimates will be more accurate; the outcome will be guaranteed upfront. Everything is always better on the second iteration. Then imagine that a DevOps Accelerator performs the same work, a dozen or more iterations per year. That's a lot of growth.

This relatively safe and proven path works well for smaller businesses whose infrastructure requirements are satisfied by the output of a sole individual. Independent Contractors are the best choice when you have a small project and know what you need. You can often negotiate better rates when working directly with a freelancer since there are no middlemen. Companies sometimes convert contractors to employees (W2) because the relationship is so successful.

It can be problematic, however, since as a business you have legally no control over when or how the Independent Contractor performs the work. They are an independent business and under the law, need to operate like a business, which means they have multiple customers. Customer demands are unpredictable, project scopes can quickly explode unexpectedly in scope, and life's surprises can pull them away from your project. The best contractors will want to please all their customers, but this can easily overwhelm them if the demands exceed their capacity.

You also must be aware of laws in your jurisdiction working with Independent Contractors. Many states are beginning to tighten their laws around working with Independent Contractors and using California as an example.

California AB5 Law

California is very strict on this under the AB5 law^[1]https://www.dir.ca.gov/dlse/faq_independentcontractor.htm. Many companies' understanding of how AB5 works are outdated, as the law was re-interpreted in 2019 ^[2]Vazquez v. Jan-Pro Franchising International, Inc. that Dynamex, which means how it is enforced is retroactive to the date the law was initially introduced. If Independent Contractors do not pass the ABC test, they may be considered employees under the law, and there's no amount of contract language that can circumvent it or prevent it. All that matters is where the work is performed; if you are a California company or the Independent Contractor is based in California, the state will claim jurisdiction. The longer you work with the Independent Contractor, the greater the risk that they will be classified as an employee. The worst part is that there are almost no consequences for the Independent Contractor, so the hiring entity bears all financial risks. There's almost no way for a business to verify that an Independent Contractor meets the “Business Services Provider” criteria of the AB5 law, and many Independent Contractors are unaware of how this law works. As of 2020, the California EDD resumed all payroll tax audits targeting the 2 million independent contractors in California ^[3]https://www.prnewswire.com/news-releases/ca-edd-confirms-it-has-resumed-tax-audits-relating-to-the-misclassification-of-10-million-contractors-301125947.html.

Working with an established DevOps Accelerator like Cloud Posse eliminates these risks. The DevOps Accelerator is responsible for complying with all local and federal employment laws. It handles all the staffing and business continuity issues and has the preexisting materials ready to perform the implementation. The hiring company can continue to focus on its core product rather than get distracted by implementing its next-generation infrastructure and platform, a non-core competency of the business. When the project completes, the hiring company can scale back on the services from the accelerator or scale back later if it needs more help. It offers the best of both forms of engagement.

Our industry is ripe with Professional Services Companies (thousands of them) that will work closely with you to implement a fully custom solution that meets exactly your requirements. That sounds great!

The first obstacle encountered is deciding on which one to go with when every one of them claims they are the industry leader. You might reach out to your AWS Account Representative to ask for some referrals, and they'll have recommendations for AWS Partners that are a good fit for your stage. Working with one of the major professional services companies might be the safest option—you'll avoid the risks of working with an independent contractor, but it's fraught with problems.

The first challenge is how to vet them and their solution. They'll almost always claim they can't show you what it looks like because their work is confidential and protected under NDA between them and their clients. That's fair, but it still puts the buyer at the disadvantage of not knowing what they're buying before they commit. So instead, the buyer will need to rely solely on Case Studies, if they even have them, which are lovely but sell a pretty picture of the outcome but not all the dirty details.

Throughout this process, you'll mostly talk with sales reps and account managers. If you're a technical organization, you'll be frustrated with their explanations' hyperbole, flashy pitch decks, and lack of technical detail. You'll have little knowledge of who you'll actually be working with, and chances are they'll take a junior engineer and mark them up 2-4x. It's a very profitable business for them, but you're left ultimately holding the bag: custom infrastructure built for you might sound great, but it's a hornet's nest to maintain in the long run. Of course, this is what they hope for so that there'll be a steady stream of follow-up work. As a result, these sorts of projects cost way more than budgeted.

While the professional services company may have a library of case studies, it doesn't mean they have a repeatable process. On top of that, they may be reimplementing everything for you unless they bring with them existing materials, which is reinventing the wheel every time; each implementation is a snowflake that will stop evolving when the contract ends. Any continued learning by the professional services company will not be passed along to you. If they bring their own materials, ensure that you know how they are licensed and that you're not on the hook for ongoing license fees.

Now contrast that experience to working with a DevOps Accelerator like Cloud Posse. The accelerator will have a proven, documented process that they follow every time, ensuring consistent results without creating unmaintainable snowflakes. They can show you precisely what you will get before you begin.

You should expect to receive live demos as part of the presales process, with clear and concise answers on what exactly you will receive — less handwaving and no fancy PowerPoint presentations. You should meet directly with highly technical engineers and skip all the b.s. sales mumbo-jumbo that makes developers roll their eyes.

Our solution is not conveniently gated behind confidentiality agreements. We use an Open Source distribution model ensuring that you do not miss out on all the continued learnings and are not on the hook for expensive ongoing license fees like other companies. It's like buying and owning a Tesla; you continually receive free updates that increase the longevity of your investment by providing free Over-the-Air (OTA) upgrades that contain bug fixes and enhancements.

Outsourcing is probably the most economical way of building out your infrastructure, but it's fraught with risk. In the risk/reward paradigm, you are rewarded for taking more significant risks that may pay off, but you need to know what you're doing.

Do not outsource your cloud architecture and its implementation to the same partner unless you have the in-house expertise to validate everything delivered to you one pull request at a time (if you're not getting pull requests? RUN!!!) Be very cautious if you don't know what you need because you may be sold something that will simply need to be redesigned at the next stage of your growth. Make sure all the work is performed in version control repositories (e.g. GitHub, GitLab, Bitbucket) that you control, with regular commits and demonstrated functionality. Don't sign off on anything until you've seen it in action. Also, be advised that it's almost impossible to conduct meaningful personnel background checks in many foreign countries, a requirement for many compliance benchmarks. Also, it's almost impossible to enforce contract terms in many foreign countries; if you need to, it will be very costly and conducted in unfamiliar jurisdictions.

On the flip side, when you work with an onshore DevOps Accelerator like Cloud Posse, you buy a known quantity – a working solution that you can vet before starting. You have the peace of mind of knowing that if anything goes seriously wrong, you have recourse; as a provider, we guarantee our work and carry sufficient insurance to do the work we perform. Cloud Posse knows exactly how our solution scales as your business evolves and Cloud Posse continually invests in the solution, which benefits the buyer over the long term.

Working with a Managed Services Provider makes sense if you're a nontechnical organization and there's no sense in understanding or knowing how your infrastructure works; your business has no competitive advantage in controlling all the toggles, and you have very few opinions on how it gets done or implemented. This is powerful when you can focus entirely on delivering your product, and you're not held up on the minutia of infrastructure.

The problem is it's “outsourcing” your competitive advantage within your industry. While it may make a lot of sense to outsource transactional areas of your business like HR and accounting, infrastructure relates to your product. When you adopt the DevOps methods, you are leveling the business up across multiple divisions, enabling you to be nimble and rapidly respond to market trends. That's why this method will fail in the long run, even with initial success. Remember, most “Case Studies” are conducted immediately after a project's implementation, the time at which it's most likely to be successful; they are not longitudinal studies on how the solution performs years later.

In the DevOps Accelerator model that Cloud Posse offers, you have all the benefits associated with working together with an MSP without the risks of outsourcing your advantage. Cloud Posse works directly with you to ensure you have everything you need to own and operate your infrastructure – including all the infrastructure as code, documentation, and processes for day-2 operations. Cloud Posse will remain engaged even after the initial scope of work is completed and provide you with the ongoing support you need until you have the operational excellence to do it all yourself. Because you own this business area, the methods become part of your processes. These processes will evolve and become the strategic advantage you need in a competitive market landscape, enabling you to phase shift and out-tack your competition.