Containerized Metricbeat/Filebeat to monitor E,K,EntSearch

[alongaks]

Hello,

I am working on getting metricbeat (and eventually filebeat ) to report on Elasticsearch, Kibana and Enterprise Search via Docker.

I'm using RHEL as the single-node host and as such it is using Podman as the Docker equivalent.

I have Elasticsearch, Kibana and Enterprise Search running fine. They come up healthy from the compose file. My issue is once I launch the Kibana UI > Stack Monitoring it shows as though there is no Metricbeat configured to see the relevant monitoring data. If I click in the UI to 'set up monitoring with Metricbeat' the Elasticsearch node appears to be offline.

I'm guessing I have something in the metricbeat configs being fed to the compose file incorrect/missing. Not sure why the Kibana portal can see the elasticsearch host container but as 'offline'.

If I enable 'self-moniitoring' w/o metricbeat from the Elasticsearch service in the compose file the monitoring shows up right away, minus Enterprise Search. However, I believe that is a long deprecated method for metric collection.

The metricbeat container log appears(?) to be pulling stats from elasticsearch and kibana OK. I will work on EntSearch some more once I can get the Elasticsearch node to show as 'online' in the Kibana UI.

{"log.level":"info","@timestamp":"2023-11-08T13:10:54.922Z","log.logger":"monitoring","log.origin":{"file.name":"log/log.go","file.line":187},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"cpuacct":{"total":{"ns":51579778}},"memory":{"mem":{"usage":{"bytes":98033664}}}},"cpu":{"system":{"ticks":230,"time":{"ms":10}},"total":{"ticks":910,"time":{"ms":40},"value":910},"user":{"ticks":680,"time":{"ms":30}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":14},"info":{"ephemeral_id":"6038f63f-e951-4ee9-a2dd-5fd3b68686c7","uptime":{"ms":181218},"version":"8.10.3"},"memstats":{"gc_next":26585296,"memory_alloc":13371752,"memory_total":169380336,"rss":90091520},"runtime":{"goroutines":99}},"libbeat":{"config":{"module":{"running":3}},"output":{"events":{"acked":45,"active":0,"batches":9,"total":45},"read":{"bytes":11092},"write":{"bytes":65413}},"pipeline":{"clients":15,"events":{"active":0,"published":45,"total":45},"queue":{"acked":45}}},"metricbeat":{"elasticsearch":{"node":{"events":3,"success":3},"node_stats":{"events":3,"success":3}},"enterprisesearch":{"health":{"events":3,"failures":3},"stats":{"events":3,"failures":3}},"kibana":{"status":{"events":3,"success":3}},"system":{"cpu":{"events":3,"success":3},"load":{"events":3,"success":3},"memory":{"events":3,"success":3},"network":{"events":9,"success":9},"process":{"events":6,"success":6},"process_summary":{"events":3,"success":3},"socket_summary":{"events":3,"success":3}}},"system":{"load":{"1":7.21,"15":1.9,"5":4.64,"norm":{"1":3.605,"15":0.95,"5":2.32}}}},"ecs.version":"1.6.0"}}

Below is the Metricbeat section of the compose file:

metricbeat:
    depends_on:
      es01:
        condition: healthy
      kibana:
        condition: healthy
      enterprisesearch:
        condition: healthy
    container_name: metricbeat
    image: docker.elastic.co/beats/metricbeat:${STACK_VERSION}
    user: root
    volumes:
      - metricbeat:/usr/share/metricbeat/data
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml:ro
      - /var/run/podman/podman.sock:/var/run/docker.sock:ro
      - certs:/usr/share/metricbeat/config/certs
    networks:
      - es-network

Below is the metricbet.yml being called:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false

processors:
  - add_cloud_metadata: ~
  - add_docker_metadata: ~

output.elasticsearch:
  hosts: ["https://es01:9200"]
  username: "elastic"
  password: "a-password-here"
  ssl.verification_mode: "none"

metricbeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

metricbeat.modules:
  - module: elasticsearch
    metricsets: ["node", "node_stats"]
    period: 10s
    hosts: ["https://es01:9200"]
    ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca/ca.crt"]
    username: "elastic"
    password: "a-password-here"
    enabled: true
  - module: kibana
    metricsets: status
    period: 10s
    hosts: https://hostname:5601
    ssl.verification_mode: none
    # I will address below once the elasticsearch node can be monitored.
    #ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/kibana.crt"]
    username: "elastic"
    password: "a-password-here"
    enabled: true
  - module: enterprisesearch
    metricsets: ["health", "stats"]
    period: 10s
    hosts: ["http://hostname:3002"]
    username: "elastic"
    password: "a-password-here"
    enabled: true

Any help/another pair of eyes is appreciated!

[alongaks]

I resorted to RPM installs in the container host of Metricbeat and Filebeat to get it to communicate with the container services of the stack.

The data shows up as expected in the Kibaba UI > Stack Monitoring.