#office-hours - July 2021

How are people in our community handling drift In the Real World? Did anyone try driftctl? What “homebrew” solutions have community members in use at the moment?

1. have a directory with yaml files each containing a single "alert"
2. in each yaml alert file have some non-environment specific values like "query" which will always be the same from cluster-to-cluster
3. have an "options" section which, basically/unfortunately, contains all the environmental specification, mainly the threshold values *for every environment*
4. loop through all these files using terraform fileset
5. yamldecode every file
6. put the name of the alert in a local tf object
7. add the non-environment specific options as a list
8. get all the environment specific stuff and add it to that object
9. iterate over all that with terraform each spinning up alerts in the process with names and environmental config 

# example
name: istio-host-latency-alert
type: metric alert
query: |
  avg(last_5m):top(top(avg:istio.mesh.request.duration.milliseconds.sum{cluster_name:${stage}} by {host}, 10, 'mean', 'desc')/top(avg:istio.mesh.request.duration.milliseconds.count{cluster_name:${eks_cluster}} by {host}, 10, 'mean', 'desc'),10,'mean','desc') > ${critical_threshold}
message: |
  ({{event.tags.cluster_name}}) Detected Host Latency Greater than 1 Second
escalation_message: ""
tags: []
options:
  - dev:
      notify_no_data: false
      renotify_interval: 60
      notify_audit: false
      timeout_h: 60
      include_tags: true
      require_full_window: true
      threshold:
        critical: 1000
        warning: 50
  - qa:
      notify_no_data: false
      renotify_interval: 60
      notify_audit: false
      timeout_h: 60
      include_tags: true
      require_full_window: true
      threshold:
        critical: 1000
        warning: 50
  - client-test:
      notify_no_data: false
      renotify_interval: 60
      notify_audit: false
      timeout_h: 60
      include_tags: true
      require_full_window: true
      threshold:
        critical: 1000
        warning: 50
...

  init:dev:
    desc: initialize tools for a developer, but not required for CI
    cmds:
      - |
        dotnet --list-sdks  || echo -e "{{ .red}} :small_red_triangle: dotnet-tools not available. Please install manually the first time here: <https://dotnet.microsoft.com/download> and then run command again {{.nocolor}}"
        mkdir -p {{ .TOOLS_DIRECTORY }} || echo -e "{{ .dark_gray}}:arrows_counterclockwise: skipped creating {{ .TOOLS_DIRECTORY }} directory per already exists {{.nocolor}}"
        # Uninstall manually
        export PATH="$PATH:{{ .HOME }}/.dotnet/tools"
        dotnet tool install --global GitVersion.Tool || dotnet tool update --global GitVersion.Tool # && echo -e "{{.green}} :white_check_mark: gitversion tool installed{{.nocolor}}"
        # If gitversion gives problems with docker commands then evaluate just running as dotnettool
        lefthook install