# Metricbeat configuration not applied correctly

**URL:** <https://discuss.elastic.co/t/metricbeat-configuration-not-applied-correctly/349873>\
**Category:** Beats\
**Tags:** docker, metricbeat\
**Created:** [December 22, 2023, 6:06pm UTC](https://discuss.elastic.co/t/metricbeat-configuration-not-applied-correctly/349873 "2023-12-22T18:06:44Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\
**Post date:** [December 22, 2023, 6:06pm UTC](https://discuss.elastic.co/t/metricbeat-configuration-not-applied-correctly/349873/1 "2023-12-22T18:06:44Z")

</div>

I am running metricbeat in a docker container and for some reasons it doesn't connect correctly to elasticsearch and logstash.

This is my container configuration:

```auto
  metricbeat:
    container_name: elk_metricbeat
    hostname: elk_metricbeat
    build:
      context: metricbeat/
      args:
        ELK_VERSION: $ELK_VERSION
    restart: always
    cap_add:
      - NET_ADMIN
      - NET_RAW
    command:
      #- /bin/bash
      #- -c
      #- while true; do metricbeat -e; sleep 1; done
      - -e
      - --strict.perms=false
      - --system.hostfs=/hostfs
    volumes:
      - ./metricbeat/config/metricbeat.yml:${METRICBEAT_DIR}/metricbeat.yml
      - metricbeatdata01:/usr/share/metricbeat/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
    environment:
      METRICBEAT_DIR: ${METRICBEAT_DIR}
      CONFIG_DIR: ${METRICBEAT_DIR}/config
      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      ELASTIC_URL: https://elasticsearch:9200
      KIBANA_URL: https://kibana:5601
      LOGSTASH_URL: http://logstash:9600
      ES_JAVA_OPTS: "-Xmx${METRICBEAT_HEAP} -Xms${METRICBEAT_HEAP}"
    secrets:
      - source: elastic-stack-ca.p12
        target: /etc/pki/ca-trust/source/anchors/elastic-stack-ca.p12
      - source: ca.crt
        target: /etc/pki/ca-trust/source/anchors/ca.crt
      - source: metricbeat.cert
        target: ${METRICBEAT_DIR}/config/metricbeat.crt
      - source: metricbeat.key
        target: ${METRICBEAT_DIR}/config/metricbeat.key
    networks:
      - elk
    depends_on:
      - kibana
      - elasticsearch

```

This is my metricbeat.yml

```auto
metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

metricbeat.config:
  reload.enabled: false
  modules.path: ${METRICBEAT_DIR}/modules.d/*.yml

metricbeat.modules:

- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
# - image
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

- module: elasticsearch
  hosts: ${ELASTIC_URL}
  metricsets:
    - node
    - node_stats
    #- index
    #- index_recovery
    #- index_summary
    #- ingest_pipeline
    #- shard
    #- ml_job
  ssl.certificate_authorities: ["/etc/pki/ca-trust/source/anchors/ca.crt"]
  ssl.certificate: config/metricbeat.crt
  ssl.key: config/metricbeat.key
  ssl.enabled: true
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  period: 10s
  xpack.enabled: true

- module: logstash
  hosts: ${LOGSTASH_URL}
  xpack.enabled: true
  period: 10s

- module: kibana
  hosts: ${KIBANA_URL}
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  xpack.enabled: true
  period: 10s
  metricsets:
    - stats

processors:
  - add_host_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: ${ELASTIC_URL}
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  ssl:
    - certificate: config/metricbeat.crt
    - certificate_authorities: ["/etc/pki/ca-trust/source/anchors/ca.crt"]
    - key: config/metricbeat.key

```

And this my error messages:

```auto
{"log.level":"info","@timestamp":"2023-12-22T17:44:29.219Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://elasticsearch:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:29.989Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9200/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:32.021Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9200/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:32.891Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9300/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:33.414Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node_stats: error making http request: Get \"http://172.29.4.7:5044/\": dial tcp 172.29.4.7:5044: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:33.868Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9300/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:37.013Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.stats: error making http request: Get \"http://172.29.4.3:5601/api/status\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-12-22T17:44:37.926Z","log.logger":"elasticsearch.ccr","log.origin":{"file.name":"ccr/ccr.go","file.line":78},"message":"the CCR feature is available with a platinum or enterprise Elasticsearch license. You currently have a basic license. Either upgrade your license or remove the ccr metricset from your Elasticsearch module configuration.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:38.360Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.status: error making http request: Get \"http://172.29.4.3:5601/api/status\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}

```

Going through these messages I see in the first info message that at least the output configuration is picked up correctly and the SSL configuration is OK.

But all these error messages trying to connect to services which are not configured anywhere in my configuration file. Where are these coming from?

I doublechecked if there are modules enabled in the _`modules.d folder`_ but the only module enabled is system and there are no hosts configured.

Also interesting is the warning message. It complains that CCR feature is enabled. I didn't find this in the module configuration at all - so where is this coming from?

I also tried to run the following commands within the container:

> metricbeat test config  
> metricbeat test modules  
> metricbeat test output

And all three basically report OK. With one exception - the elasticsearch module has an issue with the CCR and ml\_job feature. All others reported as OK.

Running `metricbeat export config` gives the following output:

```auto
metricbeat:
  autodiscover:
    providers:
    - hints:
        enabled: true
      type: docker
  config:
    modules:
      path: ${METRICBEAT_DIR}/modules.d/*.yml
    reload:
      enabled: false
  modules:
  - enabled: true
    hosts:
    - unix:///var/run/docker.sock
    metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
    module: docker
    period: 10s
  - hosts: ${ELASTIC_URL}
    metricsets:
    - node
    - node_stats
    module: elasticsearch
    password: ${ELASTIC_PASSWORD}
    period: 10s
    ssl:
      certificate: config/metricbeat.crt
      certificate_authorities:
      - /etc/pki/ca-trust/source/anchors/ca.crt
      enabled: true
      key: config/metricbeat.key
    username: ${ELASTIC_USERNAME}
    xpack:
      enabled: true
  - hosts: ${LOGSTASH_URL}
    module: logstash
    period: 10s
    xpack:
      enabled: true
  - hosts: ${KIBANA_URL}
    metricsets:
    - stats
    module: kibana
    password: ${ELASTIC_PASSWORD}
    period: 10s
    username: ${ELASTIC_USERNAME}
    xpack:
      enabled: true
output:
  elasticsearch:
    hosts: ${ELASTIC_URL}
    password: ${ELASTIC_PASSWORD}
    ssl:
    - certificate: config/metricbeat.crt
    - certificate_authorities:
      - /etc/pki/ca-trust/source/anchors/ca.crt
    - key: config/metricbeat.key
    username: ${ELASTIC_USERNAME}
path:
  config: /usr/share/metricbeat
  data: /usr/share/metricbeat/data
  home: /usr/share/metricbeat
  logs: /usr/share/metricbeat/logs
processors:
- add_host_metadata: null
- add_docker_metadata: null

```

Any ideas where I misconfigured something?

---

<div class="post-metadata">

**Author:** ![litronics](https://avatars.discourse-cdn.com/v4/letter/l/8491ac/32.png) [@litronics](https://discuss.elastic.co/u/litronics)\
**Post date:** [December 23, 2023, 8:33am UTC](https://discuss.elastic.co/t/metricbeat-configuration-not-applied-correctly/349873/2 "2023-12-23T08:33:59Z")

</div>

after playing arround with the configuration an doing some trial and error changes I found that the autodiscover feature is causing these error messages.

Interestingly - as far as I understand the configuration I only enabled it for docker only.  
Probably I can just ignore these errors, but at the end its weird somehow that they appear.

> [@litronics](#):
>
> ```auto
> metricbeat.autodiscover:
> providers:
> - type: docker
> hints.enabled: true
> 
> ```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 20, 2024, 10:34am UTC](https://discuss.elastic.co/t/metricbeat-configuration-not-applied-correctly/349873/3 "2024-01-20T10:34:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.