Metricbeat configuration not applied correctly

Elastic Stack Beats
,
1

I am running metricbeat in a docker container and for some reasons it doesn't connect correctly to elasticsearch and logstash.

This is my container configuration:

  metricbeat:
    container_name: elk_metricbeat
    hostname: elk_metricbeat
    build:
      context: metricbeat/
      args:
        ELK_VERSION: $ELK_VERSION
    restart: always
    cap_add:
      - NET_ADMIN
      - NET_RAW
    command:
      #- /bin/bash
      #- -c
      #- while true; do metricbeat -e; sleep 1; done
      - -e
      - --strict.perms=false
      - --system.hostfs=/hostfs
    volumes:
      - ./metricbeat/config/metricbeat.yml:${METRICBEAT_DIR}/metricbeat.yml
      - metricbeatdata01:/usr/share/metricbeat/data
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /proc:/hostfs/proc:ro
      - /:/hostfs:ro
    environment:
      METRICBEAT_DIR: ${METRICBEAT_DIR}
      CONFIG_DIR: ${METRICBEAT_DIR}/config
      ELASTIC_USERNAME: ${ELASTIC_USERNAME}
      ELASTIC_PASSWORD: ${ELASTIC_PASSWORD}
      ELASTIC_URL: https://elasticsearch:9200
      KIBANA_URL: https://kibana:5601
      LOGSTASH_URL: http://logstash:9600
      ES_JAVA_OPTS: "-Xmx${METRICBEAT_HEAP} -Xms${METRICBEAT_HEAP}"
    secrets:
      - source: elastic-stack-ca.p12
        target: /etc/pki/ca-trust/source/anchors/elastic-stack-ca.p12
      - source: ca.crt
        target: /etc/pki/ca-trust/source/anchors/ca.crt
      - source: metricbeat.cert
        target: ${METRICBEAT_DIR}/config/metricbeat.crt
      - source: metricbeat.key
        target: ${METRICBEAT_DIR}/config/metricbeat.key
    networks:
      - elk
    depends_on:
      - kibana
      - elasticsearch

This is my metricbeat.yml

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

metricbeat.config:
  reload.enabled: false
  modules.path: ${METRICBEAT_DIR}/modules.d/*.yml



metricbeat.modules:

- module: docker
  metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
#    - image
    - memory
    - network
  hosts: ["unix:///var/run/docker.sock"]
  period: 10s
  enabled: true

- module: elasticsearch
  hosts: ${ELASTIC_URL}
  metricsets:
    - node
    - node_stats
    #- index
    #- index_recovery
    #- index_summary
    #- ingest_pipeline
    #- shard
    #- ml_job
  ssl.certificate_authorities: ["/etc/pki/ca-trust/source/anchors/ca.crt"]
  ssl.certificate: config/metricbeat.crt
  ssl.key: config/metricbeat.key
  ssl.enabled: true
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  period: 10s
  xpack.enabled: true

- module: logstash
  hosts: ${LOGSTASH_URL}
  xpack.enabled: true
  period: 10s

- module: kibana
  hosts: ${KIBANA_URL}
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  xpack.enabled: true
  period: 10s
  metricsets:
    - stats

processors:
  - add_host_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: ${ELASTIC_URL}
  username: ${ELASTIC_USERNAME}
  password: ${ELASTIC_PASSWORD}
  ssl:
    - certificate: config/metricbeat.crt
    - certificate_authorities: ["/etc/pki/ca-trust/source/anchors/ca.crt"]
    - key: config/metricbeat.key

And this my error messages:

{"log.level":"info","@timestamp":"2023-12-22T17:44:29.219Z","log.logger":"publisher_pipeline_output","log.origin":{"file.name":"pipeline/client_worker.go","file.line":145},"message":"Connection to backoff(elasticsearch(https://elasticsearch:9200)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:29.989Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9200/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:32.021Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9200/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:32.891Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.cluster_stats: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9300/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:33.414Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset logstash.node_stats: error making http request: Get \"http://172.29.4.7:5044/\": dial tcp 172.29.4.7:5044: connect: connection refused","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:33.868Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset elasticsearch.index: error determining if connected Elasticsearch node is master: error making http request: Get \"http://172.29.4.2:9300/_nodes/_local/nodes\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:37.013Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.stats: error making http request: Get \"http://172.29.4.3:5601/api/status\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"warn","@timestamp":"2023-12-22T17:44:37.926Z","log.logger":"elasticsearch.ccr","log.origin":{"file.name":"ccr/ccr.go","file.line":78},"message":"the CCR feature is available with a platinum or enterprise Elasticsearch license. You currently have a basic license. Either upgrade your license or remove the ccr metricset from your Elasticsearch module configuration.","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"error","@timestamp":"2023-12-22T17:44:38.360Z","log.origin":{"file.name":"module/wrapper.go","file.line":256},"message":"Error fetching data for metricset kibana.status: error making http request: Get \"http://172.29.4.3:5601/api/status\": EOF","service.name":"metricbeat","ecs.version":"1.6.0"}

Going through these messages I see in the first info message that at least the output configuration is picked up correctly and the SSL configuration is OK.

But all these error messages trying to connect to services which are not configured anywhere in my configuration file. Where are these coming from?

I doublechecked if there are modules enabled in the modules.d folder but the only module enabled is system and there are no hosts configured.

Also interesting is the warning message. It complains that CCR feature is enabled. I didn't find this in the module configuration at all - so where is this coming from?

I also tried to run the following commands within the container:

metricbeat test config
metricbeat test modules
metricbeat test output

And all three basically report OK. With one exception - the elasticsearch module has an issue with the CCR and ml_job feature. All others reported as OK.

Running metricbeat export config gives the following output:

metricbeat:
  autodiscover:
    providers:
    - hints:
        enabled: true
      type: docker
  config:
    modules:
      path: ${METRICBEAT_DIR}/modules.d/*.yml
    reload:
      enabled: false
  modules:
  - enabled: true
    hosts:
    - unix:///var/run/docker.sock
    metricsets:
    - container
    - cpu
    - diskio
    - healthcheck
    - info
    - memory
    - network
    module: docker
    period: 10s
  - hosts: ${ELASTIC_URL}
    metricsets:
    - node
    - node_stats
    module: elasticsearch
    password: ${ELASTIC_PASSWORD}
    period: 10s
    ssl:
      certificate: config/metricbeat.crt
      certificate_authorities:
      - /etc/pki/ca-trust/source/anchors/ca.crt
      enabled: true
      key: config/metricbeat.key
    username: ${ELASTIC_USERNAME}
    xpack:
      enabled: true
  - hosts: ${LOGSTASH_URL}
    module: logstash
    period: 10s
    xpack:
      enabled: true
  - hosts: ${KIBANA_URL}
    metricsets:
    - stats
    module: kibana
    password: ${ELASTIC_PASSWORD}
    period: 10s
    username: ${ELASTIC_USERNAME}
    xpack:
      enabled: true
output:
  elasticsearch:
    hosts: ${ELASTIC_URL}
    password: ${ELASTIC_PASSWORD}
    ssl:
    - certificate: config/metricbeat.crt
    - certificate_authorities:
      - /etc/pki/ca-trust/source/anchors/ca.crt
    - key: config/metricbeat.key
    username: ${ELASTIC_USERNAME}
path:
  config: /usr/share/metricbeat
  data: /usr/share/metricbeat/data
  home: /usr/share/metricbeat
  logs: /usr/share/metricbeat/logs
processors:
- add_host_metadata: null
- add_docker_metadata: null

Any ideas where I misconfigured something?

2

after playing arround with the configuration an doing some trial and error changes I found that the autodiscover feature is causing these error messages.

Interestingly - as far as I understand the configuration I only enabled it for docker only.
Probably I can just ignore these errors, but at the end its weird somehow that they appear.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.