#airship - April 2019

Hi guys

I have a problem with service shown in the snippet. I deployed the service successfully earlier, but ran into problems after destroying my environment over night and reapplying it today.

After first terraform apply I get the following error:

Error: Error applying plan:

1 error(s) occurred:

* module.linkmobility.module.live_task_lookup.data.aws_lambda_invocation.lambda_lookup: data.aws_lambda_invocation.lambda_lookup: AccessDeniedException: The role defined for the function cannot be assumed by Lambda.
        status code: 403, request id: 280f95ac-e6a2-4738-935e-5b9c013e9ceb

It seems related to the timing of resource creation, but when I rerun the apply I get a new error every time I rerun:

Error: Error applying plan:

1 error(s) occurred:

* module.linkmobility.module.ecs_service.aws_ecs_service.app_with_lb: 1 error(s) occurred:

* aws_ecs_service.app_with_lb: ClientException: TaskDefinition is inactive
        status code: 400, request id: ab7ef1d7-5601-11e9-8828-d365d379c104 "linkmobility"

I think it's because the old task defintion never gets deleted, just deactivated, and that trips up the new creation.

@Maarten van der Hoef & @Maciek Strömich: Any suggestions?

Will try to get back to you this evening, have to work on work stuff now.

“* module.linkmobility.module.live_task_lookup.data.aws_lambda_invocation.lambda_lookup: data.aws_lambda_invocation.lambda_lookup: AccessDeniedException: The role defined for the function cannot be assumed by Lambda.”

interesting, let me get back to you later

While I've been banging on the module for the last few days, I've seen a couple of similar situations, where something is a dependency, but isn't available quickly enough. Re-running fixes most of them, but I've hit an impasse here.

The thing is that deleting a taskdef doesn't remove it, merely inactivates it. when you deactivate the last taskdef version, the entire taskdef becomes inactive, but it doesn't disappear.

looking at the docs i think it never disappears

what you can always do for now

is to add an env var

I’ve ran into issues where I’ve created an ECS service with the module, after that deleted it again, but the old task definitions are more or less kept in AWS. This creates a weird irregularity

I’m really looking fwd to implementing this the moment tf supports it: https://aws.amazon.com/about-aws/whats-new/2019/03/aws-fargate-and-amazon-ecs-support-external-deployment-controlle/

i tried to force a new taskdef with force_bootstrap_container_image but I'll try the env var now

so for now

container_envvars  {
       FIX = "1"
  }

well, you could support it now with a lambda, like you handle the live task check

• module.linkmobility.module.live_task_lookup.data.aws_lambda_invocation.lambda_lookup: data.aws_lambda_invocation.lambda_lookup: AccessDeniedException: The role defined for the function cannot be assumed by Lambda.

this however doesn’t explain the task definition issue you have

Thanks!

container_envvars         = {
    stamp = 1554291327
  }

... worked like a charm.

like i wrote, the first error disappears when i rerun. i think its timing related.

like maybe it takes a second for a new policy to propagate, and terraform is already trying to use it

hm shouldn’t be! 🙂

btw, i also submitted a PR to fix a bug in the lambdas. there was some python cut and paste in the error reporting that made all errors have the message "NaN" 🙂

i assume it's python, since it looked like python string building

anywho, thanks for the workaround

ah nice, thanks.

np, any time.

@Maarten van der Hoef how do you feel about https://github.com/blinkist/terraform-aws-airship-ecs-service/pull/59 ? 🙂

0.9.3: Minor bugfixing (#59)
Fixed bug in custom exception and error reporting


Added stack to custom error. Without it, error origin is lost.


Changed Python string building to JS concatenation.
string modulo string gives a NaN result every time.


Cleanup.


Respect the health check grace period variable


Statement actions MUST be lists in Terraform.

Is there a way to make environment vars for containers "valueFrom" instead of "value"?

Hi @Maarten van der Hoef. I got around to look at the drift detection you requested in the PR above. Looking at the code, I'm uncertain about how to proceed. There doesn't seem to be any special precautions for normal environment variables. What am I missing? 🙂

Looking at the lookup lambda, i can see where you get the env vars, but it doesn't look like javascript's describetaskdefintion returns the secrets as part of the container ...

I use ecs-deploy (https://github.com/silinternational/ecs-deploy) on my CI server to update the container image when the code changes.
The script fetches the running taskdef and replace the image before uploading a new task def.
However, if I run terraform to update something else, it detects a "change" and wants to downgrade to the last task defintion created with Terraform.
I'm not sure if this is related to secrets, or ecs-deploy "scrubbing off" something that Airship uses to detect the newest image.

Any ideas?

hi @Mads Hvelplund so as we cannot retrieve that from the datasource we need to do something else. You can store a hash of the combined secrets-names in a label, and compare the label for drift detection.

when your buildserver builds a new docker image, how do you deploy it? by running terraform, or using aws cli/api?

@Mads Hvelplund using ecs-deploy as well, and the drift detection always made sure that ecs-deploy could do its work

FYI: ecs-deploy doesn't support secrets unless you run with the changes from https://github.com/silinternational/ecs-deploy/pull/179
.. or rather, it only supports Fargate containers that use secrets, without the fix 🙂