Troubleshooting Kibana Dashboard Configuration for Metricbeat Modules

[MasoudLakestani]

How can I configure Kibana to display monitoring dashboards for the PostgreSQL and System modules when I've already set up these modules in the Metricbeat configuration file (metricbeat.yml )? While I can see that Elasticsearch and Logstash dashboards are visible in Kibana, and I've verified that the PostgreSQL and System data are successfully indexed (as observed in the metricbeat logs and Elasticsearch indices), the corresponding dashboards for these modules are not showing up. What could be the reason and how can I resolve this issue?

[stephenb (Stephen Brown)]

Hi @MasoudLakestani Welcome to the community

What version are you running?

Did you follow the steps in the quick start guide?

It's a good idea to always share all the configuration files

Most importantly did you run setup after you configured the modules

metricbeat setup -e

I'm a little confused by what you mean by the Kibana and Logstash dashboards Are you talking about stack monitoring?

Perhaps give us a little more overview of what you're trying to accomplish... And all the config files and perhaps we can help.

[MasoudLakestani]

Hello @stephenb,

Thank you for taking the time to respond to my query. I'd like to clarify the specific issue I'm facing. I require a robust monitoring system that can effectively monitor the following services: Elasticsearch, Logstash, Postgres, and the system itself. I've opted to use Metricbeat as the tool to accomplish this task.

To set up the environment, I'm running Elasticsearch, Kibana, and Logstash in a Docker-compose configuration. For testing purposes, I've initially disabled all security features.

Here is the Docker Compose configuration file I've used:

services:
  es01:
    build:
      context: .
    volumes:
      - blackeveryday:/usr/share/elasticsearch/data
      - ./synonym.txt:/usr/share/elasticsearch/config/synonyms/synonym.txt
      - ./stop_words.txt:/usr/share/elasticsearch/config/stop_words.txt
    environment:
      - discovery.type=single-node
      - xpack.security.enabled=false
      - ES_JAVA_OPTS=-Xms22g -Xmx22g
    networks:
      - elastic
    ports:
      - 9200:9200
    container_name: es01
    restart: always

  kib01:
    image: docker.elastic.co/kibana/kibana:8.9.0
    environment:
      - ELASTICSEARCH_HOSTS=http://es01:9200
    ports:
      - 5601:5601
    networks:
      - elastic
    container_name: kib01
    restart: always
    depends_on:
      - es01

  log01:
    image: docker.elastic.co/logstash/logstash:8.9.0
    volumes:
      - ./logstash.conf:/usr/share/logstash/pipeline/logstash.conf
      - ./logstash.yml:/usr/share/logstash/config/logstash.yml
      - ./postgresql-42.6.0.jar:/usr/share/logstash/postgres_driver/postgresql-42.6.0.jar
    environment:
      - ES_JAVA_OPTS=-Xms8g -Xmx8g
    networks:
      - elastic
    ports:
      - 9600:9600
    container_name: log01
    restart: always
    depends_on:
      - es01

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:8.9.0
    volumes:
      - /proc:/hostfs/proc:ro
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup:ro
      - /:/hostfs:ro
      - ./metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
      - ./modules.d:/usr/share/metricbeat/modules.d
      - /run/systemd/private:/run/systemd/private
      - /var/run/dbus/:/var/run/dbus/
    user: root
    container_name: metr01
    networks:
      - elastic
    depends_on:
      - es01
networks:
  elastic:
    driver: bridge
volumes:
  blackeveryday:

And here is the content of the metricbeat.yml configuration file:

metricbeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
metricbeat.max_start_delay: 40s
metricbeat.modules:
 - module: system
   metricsets:
     - cpu
     - load
     - memory
     - network
     - process
     - process_summary
   xpack.enabled: true
   enabled: true
   period: 10s
   processes: ['.*']
   cpu.metrics: ["percentages", "normalized_percentages"]
  - module: elasticsearch
    metricsets:
      - node
      - node_stats
      - index
      - index_recovery
      - index_summary
      - ingest_pipeline
      - shard
      - ml_job
    xpack.enabled: true
    period: 10s
    hosts: ["es01:9200"]

 - module: logstash
   metricsets: ["node", "node_stats"]
   xpack.enabled: true
   period: 10s
   hosts: ["log01:9600"]

 - module: postgresql
   xpack.enabled: true
   enabled: true
   metricsets:
     - database
     - bgwriter
     - activity
   period: 10s
   hosts: ["postgres://[postgres server]?sslmode=disable"]
   username: user
   password: password

output.elasticsearch:
  hosts: ["es01:9200"]

setup.kibana:
  host: "kib01:5601"
  ssl.enabled: false
setup.dashboards.enabled: true

When I navigate to the Stack Monitoring section within Kibana, I notice that only the Logstash and Elasticsearch dashboards are displayed. Even though Metricbeat appears to be successfully retrieving and indexing data for both the system and Postgres (as evidenced by the logs and Elasticsearch indices), these dashboards are conspicuously absent.

Could you provide some guidance on how I can troubleshoot this issue to ensure that the Postgres and system dashboards are displayed in Kibana?

Looking forward to your insights.

Best regards,
Masoud Lakestani

[stephenb (Stephen Brown)]

Hi @MasoudLakestani

I think there is perhaps a little basic confusion/misunderstanding

That is pretty common as there are different places to find what you are looking for.

Kibana -> Stack Monitoring
Is only for the Curated Experience for monitoring Elastic Stack Components
Think of this as a completely dedicated UI.

• Elasticsearch
• Kibana
• Logstash
• Beats
• Integration / APM Server

No other components will show up there like Postrges as they are not part of the Elastic Stack

For other components, there are 2 places to look.

For simple Infrastructure metrics you can Look at Kibana - Observability - Infrastructure or Host
This is an OOTb Curated UI Example

Then of course there are the 100s of Dashboards and Visualization for the Beat / Agents integration

So Go to Dashboard and Search for

Hosts / System Overview

or for Postgres search for Postgres I do not have that loaded right now but assuming you setup metricbeat properly A dashboard should be there

Yup a little confusing but hopefully that helps a bit.

[MasoudLakestani]

Thank you, @stephenb. My problem has been solved.