# Fleet managed elastic agent: Kubernetes Prometheus Metrics Autodiscover

**URL:** <https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808>\
**Category:** Elastic Agent\
**Tags:** metricbeat\
**Created:** [July 9, 2024, 3:12pm UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808 "2024-07-09T15:12:13Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alphayeeeet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alphayeeeet/32/126382_2.png) [@Alphayeeeet](https://discuss.elastic.co/u/Alphayeeeet)\
**Post date:** [July 9, 2024, 3:12pm UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/1 "2024-07-09T15:12:13Z")

</div>

Moved the discussion from github to here: [Prometheus Input - auto discovery and leader election in fleet · Issue #4126 · elastic/elastic-agent · GitHub](https://github.com/elastic/elastic-agent/issues/4126)

---

<div class="post-metadata">

**Author:** ![Alphayeeeet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alphayeeeet/32/126382_2.png) [@Alphayeeeet](https://discuss.elastic.co/u/Alphayeeeet)\
**Post date:** [July 9, 2024, 3:14pm UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/2 "2024-07-09T15:14:14Z")

</div>

I will just continue the discussion from Github here: @gizas Is there any possible way to include the kubernetes provider in the prometheus integration policy in fleet in the collector section?

I am not sure where to configure that provider part to get the kubernetes.\* fields.

I have no option to configure that in the UI:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/9/794ec6dbfce89d35ee214ddb69fc92ed53ef9417.png)

---

<div class="post-metadata">

**Author:** ![Andreas\_Gkizas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_gkizas/32/117035_2.png) [@Andreas\_Gkizas](https://discuss.elastic.co/u/Andreas_Gkizas)\
**Post date:** [July 10, 2024, 7:24am UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/3 "2024-07-10T07:24:46Z")

</div>

The kubernetes provider is enabled in the background.

To configure it we have this : [Advanced Elastic Agent configuration managed by Fleet | Fleet and Elastic Agent Guide [8.14] | Elastic](https://www.elastic.co/guide/en/fleet/current/advanced-kubernetes-managed-by-fleet.html#_step_2_create_a_new_configmap)

But in your case you dont need -I guess- to configure anything in the provider, just use a kubernetes variable in the condition of the policy

---

<div class="post-metadata">

**Author:** ![Alphayeeeet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alphayeeeet/32/126382_2.png) [@Alphayeeeet](https://discuss.elastic.co/u/Alphayeeeet)\
**Post date:** [July 10, 2024, 10:39am UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/4 "2024-07-10T10:39:03Z")

</div>

Thank you for the response. I will try that.

---

<div class="post-metadata">

**Author:** ![Alphayeeeet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alphayeeeet/32/126382_2.png) [@Alphayeeeet](https://discuss.elastic.co/u/Alphayeeeet)\
**Post date:** [July 10, 2024, 2:59pm UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/5 "2024-07-10T14:59:38Z")

</div>

@Andreas_Gkizas I have tried it the following way:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a91da9fe92c263325c49e49ffd674b01eb19e17a.png)

Unfortunately my agnts changed to unhealthy after deploying the policy with the following error:

[elastic\_agent][error] applying new policy: could not create the map from the configuration: error unpacking config to MapStr object: missing field accessing 'output\_permissions'

Did I configure anything wrong?

This is the API preview:

```auto
"prometheus-prometheus/metrics": {
      "enabled": true,
      "streams": {
        "prometheus.collector": {
          "enabled": true,
          "vars": {
            "hosts": [
              "${kubernetes.pod.ip}:${kubernetes.annotations.prometheus.io/port|'8090'}"
            ],
            "metrics_path": "${kubernetes.annotations.prometheus.io/path|'/metrics'}",
            "period": "30s",
            "use_types": true,
            "rate_counters": true,
            "leaderelection": true,
            "condition": "${kubernetes.annotations.prometheus.io/scrape} == \"true\"",
            "ssl.verification_mode": "none",
            "ssl.certificate_authorities": [],
            "metrics_filters.exclude": [],
            "metrics_filters.include": [],
            "headers": "# headers:\n# Cookie: abcdef=123456\n# My-Custom-Header: my-custom-value\n",
            "query": "# query:\n# key: value\n",
            "data_stream.dataset": "prometheus.collector",
            "processors": "- add_fields:\r\n target: kubernetes\r\n fields:\r\n annotations.elastic_co/dataset: ${kubernetes.annotations.elastic.co/dataset|\"\"}\r\n annotations.elastic_co/namespace: ${kubernetes.annotations.elastic.co/namespace|\"\"}\r\n annotations.elastic_co/preserve_original_event: ${kubernetes.annotations.elastic.co/preserve_original_event|\"\"}\r\n- drop_fields:\r\n fields:\r\n - kubernetes.annotations.elastic_co/dataset\r\n when:\r\n equals:\r\n kubernetes.annotations.elastic_co/dataset: \"\"\r\n ignore_missing: true\r\n- drop_fields:\r\n fields:\r\n - kubernetes.annotations.elastic_co/namespace\r\n when:\r\n equals:\r\n kubernetes.annotations.elastic_co/namespace: \"\"\r\n ignore_missing: true\r\n- drop_fields:\r\n fields:\r\n - kubernetes.annotations.elastic_co/preserve_original_event\r\n when:\r\n equals:\r\n kubernetes.annotations.elastic_co/preserve_original_event: \"\"\r\n ignore_missing: true\r\n- add_tags:\r\n tags: [\"preserve_original_event\"]\r\n when:\r\n and:\r\n - has_fields:\r\n - kubernetes.annotations.elastic_co/preserve_original_event\r\n - regexp:\r\n kubernetes.annotations.elastic_co/preserve_original_event: \"^(?i)true$\""
          }
        }

```

---

<div class="post-metadata">

**Author:** ![Alphayeeeet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alphayeeeet/32/126382_2.png) [@Alphayeeeet](https://discuss.elastic.co/u/Alphayeeeet)\
**Post date:** [July 15, 2024, 6:46am UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/6 "2024-07-15T06:46:15Z")

</div>

@Andreas_Gkizas Did I configure anything wrong in the above shared policy?

---

<div class="post-metadata">

**Author:** ![Andreas\_Gkizas](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andreas_gkizas/32/117035_2.png) [@Andreas\_Gkizas](https://discuss.elastic.co/u/Andreas_Gkizas)\
**Post date:** [July 18, 2024, 7:57am UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/7 "2024-07-18T07:57:01Z")

</div>

Hello @Alphayeeeet ,

I would say to note here the steps you did. I guess the policy is configured through Fleet right?

[quote="Alphayeeeet, post:5, topic:362808"]  
missing field accessing 'output\_permissions'  
[/quote]: The output permissions is usually needed to be able to write to appropriate indices. Are you sure you have correct privileges assigned to your user or API you try to authenticate with?

Another advise is to try step by step replacing the required fields with variables. I see you use them in hosts, metric\_path and condition  
I am not quite sure for metric\_path. Can you try just for test to put a fix string there?

Either use `./elastic-agent inspect` command to see the rendered configuration (refs [here](https://www.elastic.co/guide/en/fleet/current/elastic-agent-cmd-options.html#elastic-agent-inspect-command) and [here](https://www.elastic.co/guide/en/fleet/current/hints-annotations-autodiscovery.html#_troubleshooting))

---

<div class="post-metadata">

**Author:** ![Alphayeeeet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alphayeeeet/32/126382_2.png) [@Alphayeeeet](https://discuss.elastic.co/u/Alphayeeeet)\
**Post date:** [July 22, 2024, 11:54am UTC](https://discuss.elastic.co/t/fleet-managed-elastic-agent-kubernetes-prometheus-metrics-autodiscover/362808/8 "2024-07-22T11:54:11Z")

</div>

Hi @Andreas_Gkizas,

yes the policy is/was configured through fleet. I will configure it again by time and share the rendered config requested via diagnostics.

Still I would like to know, if there is any documentation on how to configure such condition-based autodiscovery and provider based parameteres in fleet integration policies.

As this should be quite a common usecase for that, such topic should be documented on how to use/configure it.