14 min read
Cloud Posse is the foremost DevOps Accelerator specialing in venture-backed tech startups up through F100 enterprises. This might sound like marketing hype to some, but it's our mission—it means so much more to us.
A “DevOps Accelerator” is a subcategory of Professional Services. It means that the provider has a proven, systematic, and repeatable process to help companies achieve a certain level of organizational “DevOps” maturity. The right accelerator for your business will have a similar ideology for how you would like to implement and run the technical operations of an organization. It will have demonstrated its ability to do this successfully for other businesses and will have tons of pre-existing materials that save you the cost of building it all from scratch. Some accelerators may require ongoing licenses and subscriptions, while others just give it away for free so that they can evolve at a faster rate by leveraging their community.
Working with an accelerator means you benefit from all their continuous learning across the various market segments they operate in. It means you benefit from their 20/20 hindsight—as your 20/20 foresight; you avoid all the common pitfalls associated with going it alone and instead get it right the first time and on time.
The Process
What does such a process look like? It starts with having a pre-existing project plan (e.g. in Jira) that takes a business through the journey of owning its infrastructure and includes the systems that go along with that, from beginning to end. It's much more than knowing how to write Infrastructure as Code (e.g. Terraform). It's knowing all the Design Decisions that go along with it so you can build a library of Architectural Design Records[1]https://docs.aws.amazon.com/prescriptive-guidance/latest/architectural-decision-records/architectural-decision-records.pdf. It's knowing the proper order of operations to execute it. And it's getting all the pieces to fit together across the various areas of your organization. It involves laying a solid foundation, building a platform on top of it to consistently run your services, creating a process to deliver your software consistently and reliably to that platform, and ensuring you have the observability and operational experience to drive it.
For some companies, it may also mean achieving various security benchmarks for compliance required to operate in their industry. Once this is all said and done, there needs to be a process to keep everything current and the ability to keep it up to date. There needs to exist materials for training, documentation, and knowledge transfer across the organization so that it sticks and doesn't become another failed initiative.
Our approach follows the “4-Layers of Infrastructure” model, and we always start at the bottom and build our way up to the top.
Lay a Solid Foundation
For example, for us, that means starting with your Foundational Infrastructure, which is how you manage your AWS Organization, divide up into different Organizational Units, and then segment workloads across accounts, VPCs, and subnets. A solid foundation is required for building anything, just like in the physical world. Without it, you end up with something like the Millenium Tower in San Francisco, which has a sinking problem [2]https://en.wikipedia.org/wiki/Millennium_Tower_(San_Francisco)#Sinking_and_tilting_problem. It doesn't matter how awesome or beautiful the architecture is on top of the foundation if it is at the mercy of what's beneath it.
A few of the considerations that need to be considered at the very beginning are how to operate in multiple regions, naming conventions, CIDR allocations, and so forth. Knowing your requirements for security & compliance is essential, so you get things right from the get-go, with minimal re-engineering efforts.
Build out your Platform
When you build out the platform, you decide how you want to deliver your services consistently. There are countless ways to go about this, but whatever the choice it has ramifications on how to do release engineering (e.g. CI/CD), the way applications are built, tested, and deployed, as well as how to achieve the subsequent benchmarks for compliance. A good platform will have just the right level of abstraction necessary to standardize the delivery without obfuscating how everything works and limiting the business's ability to capitalize on new trends.
A common pitfall is that companies prematurely attempt to abstract too much of their platform before they have the real-world experience to have an educated opinion on it. The wrong platform can have just as much of a detrimental effect on the business as a good one. A smaller business should be weary of emulating the platforms used by massively successful companies like Netflix and Spotify which have legions of engineers supporting them. Instead, they need to focus on a nimble platform that isn't too trendy but isn't so limited that all business value is lost.
Working with a DevOps Accelerator will ensure you get the right amount of platform for the stage you are at today. While things like a Service Mesh are tremendously advantageous for one business, it can be the Achilles Heel of the next if they don't have the in-house expertise to leverage it.
Rollout Shared Services
Once the platform is in place we're ready to deploy the shared services that are needed before we begin delivering applications on top of the platform. Businesses commonly require things like self-hosted CI/CD runners (e.g. for GitHub Actions), GitOps toolings like Atlantis or Spacelift, or observability tools like Datadog. These are part of the shared services layer because they extend the platform's capabilities.
Deploy your Applications
Modern engineering organizations are polyglots as a result of evolving with the times. They'll have something like a React frontend, with a service-oriented backend architecture consisting of Go-microservices, legacy rails apps, and maybe some high-performance rust microservices services, etc. We can never assume that the software development trends and landscape will remain the same. That's why when we work with customers to implement their Release Engineering patterns, we implement the reusable workflows and composable GitHub Actions that make build-test-and-deploy consistent and straightforward for every language, framework, and platform. We treat CI/CD as an interface for delivering your applications; therefore it must be consistent and repeatable.
Achieve Benchmark Security & Compliance
Security and Compiance is as much a technical need as a strategic business advantage. Frequently our customers want to outmaneuver their competition by achieving various benchmarks of compliance so they can land bigger deals or use it as a major differentiator when compared to their competition. Amazon understands this, which is why they offer an entire suite of security-oriented products that make achieving compliance easier. We've implemented native support for this in Terraform as part of our Security and Compliance offering which is available as part of our Reference Architecture for AWS.
Alternatives
Many more approaches are commonly taken when building out the infrastructure and platform for a business. These approaches may sound familiar, as well as the problems associated with them. Maybe the DevOps Accelerator route is for you if you've tried any of them and failed.
Build it all In-House
By in large most companies build their infrastructure using entirely in-house resources. They either pull from their existing talent pool or hire a team to build it. The advantage is that the business may think it knows exactly what it needs, so it starts building it immediately. It's a rewarding process for engineers because developers are inventive, natural creators who love to build new things.
To go this route, the business must allocate a budget, pull the resources aside to focus entirely on building out the next generation of its platform, and then construct the plan for getting there.
The risk is that the company hasn't done this before. While the team is smart and accomplished, they lack the end-to-end plan for getting there and likely underestimate the time & effort required to complete the project. In engineering, it's all too common for engineers to underestimate the level of effort required and the impact of distractions on timelines, so we overestimate the likelihood of succeeding the first time we implement anything.
Hofstadter's Law: It always takes longer than you expect, even when you take into account Hofstadter's Law.[2]
More commonly, we see that the engineers get pulled back into firefighting mode. Now the business has two problems. Maintaining the old infrastructure while attempting to innovate on the new infrastructure while exhausted from fighting fires.
Once you've built this shiny new infrastructure, it moves into the second stage: maintenance. This is a boring phase compared to the stage of building it. We see many companies hemorrhage employees, once the infrastructure is built because the fun is over; the company is left holding the bag, and the persons who built it may no longer be there to maintain it.
On the other hand, working with a DevOps Accelerator means you have the plan already in place, and the plan has been vetted and iterated on with each implementation executed by the DevOps Accelerator. This continual learning leads to a better outcome with less risk. The time estimates will be more accurate; the outcome will be guaranteed upfront. Everything is always better on the second iteration. Imagine that a DevOps Accelerator performs the same work, a dozen or more iterations per year.
Hire an Independent Contractor
This relatively safe and proven path works well for smaller businesses whose infrastructure requirements are satisfied by the output of a sole individual. Independent Contractors are the best choice when you have a small project and know what you need. You can often negotiate better rates when working directly with a freelancer since there are no middlemen. Other times, companies hire contractors directly because the relationship is so successful.
It can be problematic, however, since as a business you have legally no control over when or how the Independent Contractor performs the work. They are an independent business and under the law, need to operate like a business and have multiple customers. Customer demands are unpredictable, project scopes can easily explode unexpectedly in scope, and life's surprises can pull them away from your project. The best contractors will want to please all their customers, but this can easily overwhelm them if the demands exceed their capacity.
You also must be aware of laws in your jurisdiction working with Independent Contractors. Many states are beginning to tighten their laws around working with Independent Contractors and using California as an example.
California AB5 Law
California is very strict on this under the AB5 law[3]https://www.dir.ca.gov/dlse/faq_independentcontractor.htm. Most companies' understanding of how AB5 works are outdated, as the law was re-interpreted in 2019 [4]Vazquez v. Jan-Pro Franchising International, Inc. that Dynamex, which means how it is enforced is retroactive to the date the law was originally introduced. If Independent Contractors do not pass the ABC test, they may be considered employees under the law, and there's no amount of contract language that can circumvent it or prevent it. All that matters is where the work is performed; if you are a California company or the Independent Contractor is based in California, the state will claim jurisdiction. The longer you work with the Independent Contractor, the greater the risk that they will be classified as an employee. The worst part is that there are almost no consequences for the Independent Contractor, so the hiring entity bears all financial risks. There's almost no way for a business to verify that an Independent Contractor meets the “Business Services Provider” criteria of the AB5 law, and many Independent Contractors are unaware of how this law works. As of 2020, the California EDD resumed all payroll tax audits targeting the 2 million independent contractors in California [5]https://www.prnewswire.com/news-releases/ca-edd-confirms-it-has-resumed-tax-audits-relating-to-the-misclassification-of-10-million-contractors-301125947.html.
Working with an established DevOps Accelerator eliminates these risks. The DevOps Accelerator is responsible for complying with all local employment laws. It handles all the staffing and business continuity issues and has the preexisting materials ready to perform the implementation. The hiring company can continue to focus on its core product rather than get distracted by implementing its next-generation infrastructure and platform, a non-core competency of the business. When the project completes, the hiring company can scale back on the services from the accelerator or scale back later if it needs more help. It offers the best of both forms of engagement.
Partner with a DevOps Professional Services Company
Our industry is ripe with Professional Services Companies (thousands of them) that will work closely with you to implement a fully custom solution that meets exactly your requirements. The first obstacle encountered is deciding on which one to go with when every one of them claims they are the industry leader. You might reach out to your AWS Account Representative to ask for some referrals, and they'll have recommendations for AWS Partners that are a good fit for your stage. Working with a major professional services company might be the safest option—you'll avoid the risks of working with an independent contractor, but it's fraught with problems.
The first challenge is how to vet them and their solution. They'll almost always say they can't show you what it looks like because their work is confidential and protected by an NDA between them and their clients. That's fair, but it still puts the buyer at the disadvantage of not knowing what you're buying before you commit. So instead, you'll need to rely on Case Studies, if they even have them, which are wonderful but sell a pretty picture of the outcome but not all the dirty details.
Throughout this process, you'll mostly talk with sales reps and account managers. You'll be frustrated with their explanations' hyperbole, flashy pitch decks, and lack of technical detail if you're a technical organization. You'll have little knowledge of who you'll actually be working with, and chances are they'll take a junior engineer and mark them up 2-4x. It's a profitable business for them, but you're left ultimately holding the bag: custom infrastructure built for you might sound great, but it's a hornet's nest to maintain in the long run. Of course, this is what they hope for so that there'll be a steady stream of follow-up work. As a result, these sorts of projects cost way more than budgeted.
While the professional services company may have a library of case studies, it doesn't mean they have a repeatable process. On top of that, they may be reimplementing everything for you unless they bring materials, which is reinventing the wheel every time; each implementation is a snowflake that will stop evolving when the contract ends. Any continued learning by the professional services company will not be passed along to you.
Contrast that with working with a DevOps Accelerator. The accelerator will have a proven, documented process that they follow every time, ensuring consistent results without creating unmaintainable snowflakes. They can show you exactly what you will get before you begin.
You should expect to receive live demos as part of the presales process, with clear and concise answers on what exactly you will receive — less handwaving and no fancy PowerPoint presentations. You should meet directly with highly technical engineers and skip all the b.s. sales mumbo-jumbo that makes developers roll their eyes.
Their solution will not be conveniently gated behind confidentiality agreements. The best accelerators will leverage large amounts of Open Source as a licensing and distribution model ensuring that you do not miss out on all the continued learnings and are not on the hook for expensive ongoing license fees. It's like buying and owning a Tesla; you continually receive free upgrades that increase the longevity of your investment by providing free Over-the-Air (OTA) upgrades that contain bug fixes and enhancements.
Outsource It
Outsourcing is probably the most economical way of building out your infrastructure, but it's fraught with risk. In the risk/reward paradigm, you are rewarded for taking bigger risks when they pay off, but you need to know what you're doing. Do not outsource your cloud architecture and its implementation to the same partner unless you have the in-house expertise to validate everything delivered to you one pull request at a time. Be very cautious if you don't know what you need because you may be sold something that will simply need to be redesigned at the next stage of your growth. Make sure all the work is performed in repositories you control, with regular commits and demonstrated functionality. Don't sign off on anything until you've seen it in action. Also, be advised that it's almost impossible to conduct meaningful background checks in many foreign countries. It's almost impossible to enforce contract terms; if you need to, it will be very costly and conducted in unfamiliar jurisdictions.
On the flip side, when you work with an onshore DevOps Accelerator, you buy a known quantity – a working solution that you can vet before starting. You have the peace of mind of knowing that if anything goes seriously wrong, you have recourse; the provider should guarantee their work and carry sufficient insurance to do the work they perform. The services provider will know how their solution scales as your business evolves and will continually invest in the solution, which benefits the buyer over the long term.
Use a Managed Services Provider (MSP)
Working with a Managed Services Provider makes sense if you're a nontechnical organization and there's no sense in understanding or knowing how your infrastructure works. Your business has no competitive advantage in controlling all the toggles, and you have very few opinions on how it gets done or implemented. This is powerful when you can focus entirely on delivering your product, and you're not held up on the minutia of infrastructure.
The problem is that it's like “outsourcing” what should be your industry competitive advantage. While it may make a lot of sense to outsource transactional areas of your business like HR and accounting, infrastructure relates to your product. When you adopt the DevOps methods, you are leveling the business up across multiple divisions, enabling you to be nimble and rapidly respond to market trends. That's why this method will fail in the long run, even with initial success. Remember, most “Case Studies” are conducted immediately after a project's implementation, the time at which it's most likely to be successful; they are not longitudinal studies on how the solution performs years later.
In the DevOps Accelerator model, you have the benefits associated with working together with an MSP without the risks of outsourcing your advantage. They will work directly with you to ensure you have everything you need to own and operate your infrastructure – including infrastructure as code, documentation, and processes for day-2 operations. They will remain engaged even after the initial scope of work is completed and provide you with the ongoing support you need until you have the operational excellence to do it all yourself. Because you own this business area, the methods become part of your processes. These will evolve and become the strategic advantage you need in a competitive market landscape, enabling you to phase shift and out-tack your competition.
Next Steps
If you're curious about what benefits may await you working with a DevOps Accelerator, we encourage you to reach out. You can expect a refreshingly different experience. No hungry sales reps. Just an honest review of your problems with an experienced, highly technical cloud architect who can demonstrate with live examples how we can address them using our proven process.
References
| ↑1 | https://docs.aws.amazon.com/prescriptive-guidance/latest/architectural-decision-records/architectural-decision-records.pdf |
|---|---|
| ↑2 | https://en.wikipedia.org/wiki/Millennium_Tower_(San_Francisco)#Sinking_and_tilting_problem |
| ↑3 | https://www.dir.ca.gov/dlse/faq_independentcontractor.htm |
| ↑4 | Vazquez v. Jan-Pro Franchising International, Inc. that Dynamex |
| ↑5 | https://www.prnewswire.com/news-releases/ca-edd-confirms-it-has-resumed-tax-audits-relating-to-the-misclassification-of-10-million-contractors-301125947.html |
| ↑6 | https://the-innovator.club/index.php/2020/11/09/what-happens-when-what-you-deliver-is-not-what-the-customer-expected/ |