#prometheus - November 2022

i'm using the kube-prometheus-stack chart and i'm trying to get my pods labels into the metrics.

see my pod labels below.

Labels:       app=xx-failing-nginx
              <http://app.kubernetes.io/instance=xx-failing-nginx|app.kubernetes.io/instance=xx-failing-nginx>
              <http://app.kubernetes.io/name=xx-failing-nginx|app.kubernetes.io/name=xx-failing-nginx>
              <http://xx.net/dataclassification=confidential|xx.net/dataclassification=confidential>
              <http://xx.net/environment=uat|xx.net/environment=uat>
              <http://xx.net/networkposition=internal|xx.net/networkposition=internal>
              <http://xx.net/owner=edge|xx.net/owner=edge>
              <http://xx.net/priority=P3|xx.net/priority=P3>
              <http://xx.net/product=hft|xx.net/product=hft>
              <http://xx.net/service=xx-failing|xx.net/service=xx-failing>
              env=uat
              <http://helm.sh/chart=xx-service-0.0.33|helm.sh/chart=xx-service-0.0.33>

i added the below to the values file for the chart, in order to bring all of the labels from the pod into the metrics.

kube-state-metrics:
  prometheus:
    monitor:
      relabelings:
        - action: labelmap
          regex: __meta_kubernetes_pod_label_(.+)

but this hasn't worked...the only difference it's made is the below:

kube_pod_container_status_waiting_reason{app_kubernetes_io_component="metrics", app_kubernetes_io_instance="kube-prometheus-stack", app_kubernetes_io_managed_by="Helm", app_kubernetes_io_name="kube-state-metrics", app_kubernetes_io_part_of="kube-state-metrics", app_kubernetes_io_version="2.6.0", container="app", endpoint="http", helm_sh_chart="kube-state-metrics-4.20.2", instance="10.1.1.xx:xx", job="kube-state-metrics", namespace="uat", pod="xx-failing-nginx-5cb58d8977-8r46t", pod_template_hash="646b9498dd", reason="CrashLoopBackOff", release="kube-prometheus-stack", service="kube-prometheus-stack-kube-state-metrics"}

it seems to have added only labels and the values from the kube-state-metrics service monitor instead of the labels from my pod above.