#security - February 2020

Nice write up @lvh https://latacora.micro.blog/2020/02/19/stop-using-encrypted.html

Thanks but that’s Thomas’ writing :)

What lightweight process/tool do you guys use for Change Management, if any?

@UUB28NLDS can you keep tabs? 😃

Does nobody use change management? Or did I ask in the wrong channel? 😉

I mean, I have "change management." All my infra which includes IAM policies, security groups, accounts, etc. is defined by code. That code can't be rolled out except after reviewed PRs. So, I guess PRs are my change management. But nothing specific for the task, though I'd call that pretty light weight. It's not all encompassing, and what you're managing change on matters quite a bit too. I know nothing of specific change management tools.

Thanks. I knew someone would mention infra-as-code, but I have doubts whether it's sufficient

I do the same. Git and PRs and required status checks, with a build system managing the deploy to controlled environments. Can be as lightweight or as complex as you like

@Igor I whipped up some thoughts on this here: (draft)

https://cloudposse.com/change-management/

also welcome feedback from everyone. I'll update it with your recommendations.

@Erik Osterman (Cloud Posse) Thanks for that! So I assume (from #security perspective), there are literally no exceptions in this case. (i.e. If you urgently need to add a route to the ALB, it has to be done via TF.) In this case, shouldn't all AWS IAM accounts for humans be strictly read-only?

That's a good real-world practical example.

@marcinw has an interesting solution for this in his product, basically a way to run audited ad-hoc commands.

I don't know if there's a general-catch-all solution to address this.

For example, recorded session logs in teleport is one way

or recorded RDP sessions is another

I think the key is to have traceability for changes in addition to the control. Obviously cloudtrails will give a precise log of what changed and by whom. But ideally, every change is traced back to source control and every change in source control is traced to some ticket/task.

That way you can justify every change

Yup. Not only audited, but also guarded by Rego policies:

https://docs.spacelift.io/concepts/task
https://docs.spacelift.io/concepts/policy/task-run-policy

Also, if you implement IaC, you want drift detection of some kind, like with terraform plan. So even if you do allow someone to, say, manually add a route to the ALB in an emergency, it then starts causing your drift detection to fail until someone adds the route to the codebase also

Technically speaking terraform plan doesn’t fail, it just shows outstanding changes.

terraform plan -detailed-exitcode with exit code 2

I’m aware of that.

Haven’t thought of using it to detect drift. I guess one would run it recurrently in CI?

yep. on a schedule, or get fancy and run it in response to cloudtrail events

Wouldn’t it keep throwing false positives all over the place?

Like, you’re running your drift detection in the middle of a long apply?

the lock should prevent that

But your CI job would fail if it doesn’t acquire the lock, no?

So you still get a false positive, but a wee bit earlier.

you can test the exit code. if it's not 2 then it's not an error this CI job cares about

or you can attempt to check the lock first, but that will be subject to a race condition

What about changes waiting for human review? Or do you automatically apply everything that’s merged to your tracked branch?

automatically apply on some condition, yes. it can be a merge to a tracked branch, or a tag, or... pick your condition

OK, then that’s a bit unsafe, no? Because unless you actually apply from a saved and reviewed plan, there’s no guarantee that what you’re applying is what you saw?

depends on the size of the config and the state, and how quickly everything is moving

Yup, and how many people are contributing.

you can generate a plan, store that as an artifact, and promote it through your pipeline

Drift detection is something I’d probably want to have in Spacelift at some point.

on merge, have the ci retrieve the plan and apply that specifically

i don't mean to say that terraform plan is a perfect answer, nor the right answer for everyone, but i think it's a reasonable 80% answer that is low-drag and works for us