#kubernetes - October 2019

i am using terraform-aws-elasticsearch module in my stack and im loved it

Does sweetops has any terraform module to create Kubernetes cluster using kops?

No they use kops from the cli to provision kubernetes.

I got a tricky one for you peeps.. At a high level, I need a static IP (Elastic IP) in front of a k8s service or ing.

aws-alb-ingress-controller doesn't help since ALBs can't use EIPs out of the box.. (yes, you can put an NLB in front of it.. and have a lambda function keep the NLB target group up to date the ALB IPs.. https://aws.amazon.com/blogs/networking-and-content-delivery/using-static-ip-addresses-for-application-load-balancers/)

Using nlb annotations in a svc is feature poor even with the latest version of EKS (k8s 1.14) and doesn't properly attach EIPs to the NLB.

What else should I look at? Things that sound nice but I've never touched before (CRDs, Operators, etc..) could maybe help.. or not? What do you think?

Alternatively.. I can use terraform to stand up an NLB + EIPs.. then use a lambda function or some code somewhere to constantly update the NLB target group with the results from kubectl get nodes

This should be possible today using simple nginx ingress with the right annotations

service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/aws-load-balancer-eip-allocations: "eipalloc-07e3afcd4b7b5d644,eipalloc-0d9cb0154be5ab55d,eipalloc-0e4e5ec3df81aa3ea"

Ah so need to run a newer version of k8s not supported by eks

it is so much easier to use global accelerator

@PePe Amengual @Cameron Boulton I love you

like.. project is done already..

global accelerator is amazing

Yea, 80% of infra solutions are like this: people fall back on what they know and build these Rube Goldberg machines that have already been solved.

I'm glad it worked for you

Hey all, not sure if this belongs in this channel so please let me know if it's not the place, but I just opened up a neat feature PR for the cloudposse/prometheus-to-cloudwatch app - if anyone uses that and has some time to give some feedback I would really appreciate it, thanks! https://github.com/cloudposse/prometheus-to-cloudwatch/pull/28

@Andriy Knysh (Cloud Posse) will review ☝️

@Austin Cawley-Edwards thanks for the contribution

Awesome, thank you both!

cross posting from #security because it is relevant here: https://sweetops.slack.com/archives/CBXSAR45B/p1570720099000200

how to encrypt passwords in helm values.yaml, any good documents is appreciated. Thanks

@AG there's the helm-secrets plugin that tries to address this

but secrets will still be clear-text in the if you run helm get values

(which is why you just can't pass any secrets via helm that you truly care about)

instead, the better pattern is to assume the secrets have been installed some other way.... basically assume the resource already exists and don't provision with helm

then when you install the chart release, it will block until that secret exists.

there are a few strategies for populating secrets

basically, you want to decouple the lifecycle of secrets with the lifecycle of helm releases

@Erik Osterman (Cloud Posse) Thanks

I’m trying to pass encrypted values to secrets and use them as variables, will that work?

{{ (tpl (.Files.Glob “configs/*“).AsSecrets . ) | indent 2 }}

Hey all, trying to set up kops in a new environment set up with the reference-architectures repo, so right now trying to run kops-aws-platform (https://github.com/cloudposse/terraform-root-modules/tree/master/aws/kops-aws-platform) and it seems it expects IAM roles like masters.us-west-2.testing.ryanjarv.sh and nodes.us-west-2.testing.ryanjarv.sh to be set up. Wondering if there is some step I missed that handles that.

those are provisioned by kops

just so there's no confusion we're not using the terraform mode for kops

there are some other modules out there by others that do that

our module is for setting up the aws integration points that kops expects.

Terraform mode? Suppose I don’t know much to much about managing kops/k8s. Is that just managing individual pods with terraform? k8s in general still gets set up with the kops-aws-platform module right?

Edit: ok nvm seems the cluster itself is set up with kops.

this is what I was referring to

then there are some other terraform modules (not by us) that leverage this (i think)

interested in thoughts - my thoughts are it sounds like it's trying to separate dev and ops which i do not like

is this like https://www.docker.com/blog/announcing-cloud-native-application-bundle-cnab/?

sort of - it seems more like a way to implement an abstraction layer between teams of dev/ops/infra teams. cnab feels like more of a packaging tool kit to me, where this feels more like enterprise service catalogish kind of stuff (insert hand-waving)

while i understand the pain that's driving the need, i'm not sure i'd like to deal with an environment where that was required

i'm also a little sick of abstractions over the kube apis that just look like the kube apis 😖

Hey #kubernetes !

Just deployed k8s via the k8s-workers module, everything is working great. Being able to add iam users and roles via terraform is amazing.

Attempting to deploy a gitlab helm chart results in

Error creating load balancer (will retry): failed to ensure load balancer for service default/gitlab-nginx-ingress-controller: could not find any suitable subnets for creating the ELB

I used CloudPosse's VPC, Subnets, EKS, local.tag and EKS Workers modules

I figured it out

I needed to add the var.tags to the subnet module

@Brandon Shutter thanks! Have you looked at this working example https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf

and test for the example https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/test/src/examples_complete_test.go

I believe you are talking about these tags https://github.com/cloudposse/terraform-aws-eks-cluster/blob/master/examples/complete/main.tf#L19 (shared is required by EKS)

Hi guys,

Have just installed AWS EKS + autoscaller.
All seem to be good except autoscaler failing with the error as follow:

E1021 18:40:49.320402       1 aws_manager.go:148] Failed to regenerate ASG cache: cannot autodiscover ASGs: RequestError: send request failed
caused by: Post <https://autoscaling.eu-west-2.amazonaws.com/>: dial tcp: i/o timeout
F1021 18:40:49.320431       1 aws_cloud_provider.go:330] Failed to create AWS Manager: cannot autodiscover ASGs: RequestError: send request failed
caused by: Post <https://autoscaling.eu-west-2.amazonaws.com/>: dial tcp: i/o timeout

Not sure why it can’t reach internal AWS’s API service.
Autoscaller has been successfully installed using helm. Hence there is connectivity on the worker node.
Any advices of what else shall I check?

does anyone have an elegant solution to applying the stupid eks aws-auth config map via terraform without using a public endpoint on eks (and without being inside the vpc)? - i'm pretty sure this is pretty much technically impossible

using atlantis running in the vpc (or peer vpc), you can accomplish it.

we run atlantis inside of ECS fargate for this reason

but if the requirement is to apply it without being inside and without being outside, maybe look into aws ssm agent?

yeh - it's a frustrating requirement in that i want to be able to stand up the environment and hook up roles so that things within that environment can manage itself and connect everything - but i can't set up access to the cluster without being able to connect to the cluster.
it would be nice if eks could bootstrap the rbac config on cluster creation or you could pass through a cluster admin role arn rather than just granting system:master to the user that created the cluster

hopefully that's on the roadmap somewhere

Thanks @Erik Osterman (Cloud Posse) for the invite 🙂

Welcome @Jord!

Hey everyone! @Jord has a really neat product for learning kubernetes.

Clearly a lot of thought has gone into this.

🙂 🙏 Thanks for the shout out - if you have any Qs just DM me or mail me at jord@msb.com

Hi I need a help on creating configmap.
resource "kubernetes_config_map" "env" {
metadata {
name ="tf-${var.project}-${var.component}-env"
namespace = "${var.namespace}"

labels = {
app = "tf-${var.project}-${var.component}"
}
}

data = {
MINIO_ACCESS_KEY="minio"
MINIO_SECRET_KEY="minio123"
}

}

In the above I want to declare the values of data as variable and change it as per environment

.I am not able to declare it as string. Can someone please assist

variable "env_values" {
type = string

}
env_values = "MINIO_ACCESS_KEY=\"minio\" \nMINIO_SECRET_KEY=\"minio123\""

I tried many possible combination but nothing works
I tried using a file to declare all env variables and it worked but Minio is not picking the username in that way

Kindly give a suggestion