Metricbeat - Setup sample dashboards to the Kibana index failing

dmingueza · May 29, 2024, 2:10pm

I have Metricbeat, Kibana and Elasticsearch running in different Docker containers in the same network.

Metricbeat is sending information successfully to Elasticsearch. However, I have a problem when I want to load the sample dashboards to the Kibana index. If I execute the command: ./metricbeat setup --dashboards in Metricbeat container, I get the following error:

Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana:5601/api/status fails: fail to execute the HTTP GET request: Get "https://kibana:5601/api/status": http: server gave HTTP response to HTTPS client (status=0). Response:

My configuration is the following:

Docker Compose:

services:
#...
  kibana:
    image: docker.elastic.co/kibana/kibana:${ELK_STACK_VERSION}
    container_name: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibana_data:/usr/share/kibana/data
    env_file:
      - .kibana.env
    ports:
      - ${KIBANA_PORT}:5601
    networks:
      - elastic

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:${ELK_STACK_VERSION}
    user: root
    container_name: metricbeat
    volumes:
      - certs:/usr/share/metricbeat/config/certs
      - metricbeat_data:/usr/share/metricbeat/data
      - ./metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
      - /proc:/hostfs/proc
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup
      - /:/hostfs
    env_file:
      - .metricbeat.env
    networks:
      - elastic

.metricbeat.env

# Metricbeat
METRICBEAT_USERNAME=
METRICBEAT_PASSWORD=

# Elasticsearch
ELASTICSEARCH_URL=https://elasticsearch:9200

# Kibana
KIBANA_URL=kibana:5601
KIBANA_USERNAME=
KIBANA_PASSWORD=

.kibana.env

# Kibana
SERVERNAME=kibana

# Elasticsearch
ELASTICSEARCH_URL=https://elasticsearch:9200
ELASTICSEARCH_HOSTS=https://elasticsearch:9200
ELASTICSEARCH_USERNAME=
ELASTICSEARCH_PASSWORD=
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt

metricbeat.yml

#============================== Kibana =====================================

# Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.
# This requires a Kibana endpoint configuration.
setup.kibana:

  # Kibana Host
  # Scheme and port can be left out and will be set to the default (http and 5601)
  # In case you specify and additional path, the scheme is required: http://localhost:5601/path
  # IPv6 addresses should always be defined as: https://[2001:db8::1]:5601
  host: "${KIBANA_URL}"

  # Optional protocol and basic auth credentials.
  protocol: "https"
  username: "${KIBANA_USERNAME}"
  password: "${KIBANA_PASSWORD}"

  # Kibana Space ID
  # ID of the Kibana Space into which the dashboards should be loaded. By default,
  # the Default Space will be used.
  #space.id:

  # Use SSL settings for HTTPS.
  ssl.enabled: true

  # List of root certificates for HTTPS server verifications
  ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca/ca.crt"]

I believe there is a problem with HTTPS configuration, there is an HTTP request to Kibana API to extract the current version when the client expects to be HTTPS.

The problematic function is here: https://github.com/elastic/beats/blob/main/metricbeat/module/kibana/kibana.go#L99https://github.com/elastic/beats/blob/main/metricbeat/module/kibana/kibana.go#L99

Moreover, I have found that if I change the protocol to HTTP in Metricbeat configuration, I don't get this error but I get a 400 Bad Request in Kibana logs. Surprisingly, the dashboards get loaded in Kibana anyway. Could it be a security issue?

Gelinski · June 12, 2024, 4:17pm

Can you try again changing the KIBANA_URL to https://kibana:5601?

dmingueza · June 18, 2024, 10:39am

Same error is happening!

Loading dashboards (Kibana must be running and reachable)
Exiting: error connecting to Kibana: fail to get the Kibana version: HTTP GET request to https://kibana:5601/api/status fails: fail to execute the HTTP GET request: Get "https://kibana:5601/api/status": http: server gave HTTP response to HTTPS client (status=0).

You can see in the detail that HTTPS is being ignored. There is no difference because specifying protocol: https in settings is the same as having the protocol in the URL.

rtwolfe94022 · June 19, 2024, 1:51pm

Official Elastic Search 8.14 + Docker 4.30 Repo via JARaaS Hybrid RAG - Documentation - code current 6/19/2024 [AI Generated]

Note: Sources at the end of the response

The issue you're encountering arises from a mismatch between the protocol specified in your Metricbeat configuration and the Docker Compose environment variables. Specifically, your Metricbeat configuration (metricbeat.yml) specifies that Kibana should be accessed via HTTPS, but your environment variable KIBANA_URL does not include the protocol, which defaults to HTTP.

Here’s how you can resolve this issue:

Correct the KIBANA_URL in your environment variables to use HTTPS:

Update the .metricbeat.env file to include the HTTPS protocol:
```
# Kibana
KIBANA_URL=https://kibana:5601
```

Ensure that Kibana is set up to use HTTPS:

You need to configure Kibana to support HTTPS by modifying the kibana.yml configuration file. Here is a basic example configuration snippet for HTTPS:

server.host: "0.0.0.0"
server.ssl.enabled: true
server.ssl.certificate: /usr/share/kibana/config/certs/kibana.crt
server.ssl.key: /usr/share/kibana/config/certs/kibana.key
elasticsearch.hosts: ["https://elasticsearch:9200"]
elasticsearch.ssl.certificateAuthorities: /usr/share/kibana/config/certs/ca.crt
elasticsearch.username: "${ELASTICSEARCH_USERNAME}"
elasticsearch.password: "${ELASTICSEARCH_PASSWORD}"

Ensure Elastic's CA certificate is correctly copied and referenced in both Kibana and Metricbeat:

Make sure that the certificate files are correctly mounted inside the Docker containers and are correctly referenced in your metricbeat.yml and kibana.yml.
Restart the Docker containers:

After making the changes, restart your Docker containers to ensure that the updated configurations are applied.

If the errors persist even after verification, the detailed logging from Metricbeat and Kibana will provide further insights. Ensure your Docker containers' logs are being monitored for any further configuration hints.

Example Docker Compose Configuration:

Here's a narrowed down version of your Docker Compose services with HTTPS settings applied:

services:
  kibana:
    image: docker.elastic.co/kibana/kibana:${ELK_STACK_VERSION}
    container_name: kibana
    volumes:
      - certs:/usr/share/kibana/config/certs
      - kibana_data:/usr/share/kibana/data
      - ./kibana/config/kibana.yml:/usr/share/kibana/config/kibana.yml
    env_file:
      - .kibana.env
    ports:
      - ${KIBANA_PORT}:5601
    networks:
      - elastic

  metricbeat:
    image: docker.elastic.co/beats/metricbeat:${ELK_STACK_VERSION}
    user: root
    container_name: metricbeat
    volumes:
      - certs:/usr/share/metricbeat/config/certs
      - metricbeat_data:/usr/share/metricbeat/data
      - ./metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
      - /proc:/hostfs/proc
      - /sys/fs/cgroup:/hostfs/sys/fs/cgroup
      - /:/hostfs
    env_file:
      - .metricbeat.env
    networks:
      - elastic

networks:
  elastic:
    driver: bridge

volumes:
  certs:
    driver: local
  kibana_data:
    driver: local
  metricbeat_data:
    driver: local

Configuration Snippets:

metricbeat.yml

setup.kibana:
  host: "${KIBANA_URL}"
  protocol: "https"
  username: "${KIBANA_USERNAME}"
  password: "${KIBANA_PASSWORD}"
  ssl.enabled: true
  ssl.certificate_authorities: ["/usr/share/metricbeat/config/certs/ca/ca.crt"]

.metricbeat.env

# Metricbeat
METRICBEAT_USERNAME=
METRICBEAT_PASSWORD=

# Elasticsearch
ELASTICSEARCH_URL=https://elasticsearch:9200

# Kibana
KIBANA_URL=https://kibana:5601
KIBANA_USERNAME=
KIBANA_PASSWORD=

.kibana.env

# Kibana
SERVERNAME=kibana

# Elasticsearch
ELASTICSEARCH_URL=https://elasticsearch:9200
ELASTICSEARCH_HOSTS=https://elasticsearch:9200
ELASTICSEARCH_USERNAME=
ELASTICSEARCH_PASSWORD=
ELASTICSEARCH_SSL_CERTIFICATEAUTHORITIES=config/certs/ca/ca.crt

These adjustments should allow Metricbeat to successfully load dashboards to the Kibana index over an HTTPS connection.

Sources:

Security and monitoring setups: security-basic-setup-https.asciidoc (internal document)
General settings and configurations: monitoring-settings.asciidoc and configuring-metricbeat.asciidoc (internal documents)

dmingueza · July 12, 2024, 7:09am

Thank you very much!!

The problem wasn't related to the mismatch in Metricbeat configuration, in fact it was good because the protocol is appended to the host.

The solution is to configure Kibana to run with HTTPS, that is the step I was missing. Using the file you provided (kibana.yml) and making some adjustments to Nginx, I could make it work!

Topic		Replies	Views
Unable to load Kibana Dashboards via MetricBeat Beats metricbeat	9	9377	March 12, 2019
Metricbeat & docker & kibana dashboard Beats docker , metricbeat	3	616	July 25, 2020
Metricbeat to elasticsearch (same host with docker) connection failed [forsaken] Beats metricbeat	2	1360	December 18, 2018
Unable to setup filebeat dashboard on kibana Beats docker , filebeat	4	496	May 2, 2022
Beat not waiting for elasticsearch when sending dashboard Beats	4	959	October 4, 2018

Metricbeat - Setup sample dashboards to the Kibana index failing

Example Docker Compose Configuration:

Configuration Snippets:

Sources:

Related topics