Metricbeat on MySQL docker

[Martin_Kirouac]

Hi. As a learning experience, I am trying to install Metricbeat on the MySQL 5.7 docker official (debian flavor see Dockerfile below) image using the official ELK image (elasticsearch:7.5.1 and kibana:7.5.1). I am struggling with an Kibana dashboard error and I was hoping someone could help as I am stuck. Thanks.

root@b6fd4a1ece3d:/usr/share/metricbeat# ./bin/metricbeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Skipping loading dashboards, No directory /usr/share/metricbeat/bin/kibana/7
root@b6fd4a1ece3d:/usr/share/metricbeat# /usr/share/metricbeat/bin/kibana/7
bash: /usr/share/metricbeat/bin/kibana/7: No such file or directory

Okay, so to investigate I created that directory myself :

root@b6fd4a1ece3d:/usr/share/metricbeat# mkdir /usr/share/metricbeat/bin/kibana/7
root@b6fd4a1ece3d:/usr/share/metricbeat# ./bin/metricbeat setup
Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Skipping loading dashboards, The directory /usr/share/metricbeat/bin/kibana/7 does not contain the index-pattern or dashboard subdirectory. There is nothing to import into Kibana.

This is my metricbeat.yml :

#========================== Modules configuration ============================
metricbeat.config.modules:

Glob pattern for configuration loading

path: /etc/metricbeat/modules.d/*.yml

Set to true to enable config reloading

reload.enabled: false

Period on which files under path should be checked for changes

reload.period: 10s
#==================== Elasticsearch template setting ==========================
setup.template.settings:
index.number_of_shards: 1
index.codec: best_compression
#_source.enabled: false
#============================== Dashboards =====================================

These settings control loading the sample dashboards to the Kibana index. Loading

the dashboards is disabled by default and can be enabled either by setting the

options here, or by using the -setup CLI flag or the setup command.

setup.dashboards.enabled: true

logging.level: warning
logging.to_files: true
logging.to_syslog: false
logging.files:
path: /var/log/metricbeat
name: metricbeat.log
keepfiles: 2
permissions: 0644

#============================== Kibana =====================================

Starting with Beats version 6.0.0, the dashboards are loaded via the Kibana API.

This requires a Kibana endpoint configuration.

setup.kibana.host: "kibana:5601"
setup.kibana.protocol: "http"

output.elasticsearch:
hosts: ["elasticsearch:9200"]

metricbeat.modules:

• module: mysql
  enabled: true
  metricsets: ["status"]
  period: 5s
  hosts: ["tcp(localhost:3306)/"]
  username: user
  password: password

AND this is my Dockerfile :

Pull the mysql:5.6 image

FROM mysql:5.7

The maintainer name and email

MAINTAINER MyEmail myemail@gmail.com

ENV MYSQL_DATABASE='test'
ENV MYSQL_USER='user'
ENV MYSQL_PASSWORD='password'
ENV MYSQL_ROOT_PASSWORD='password'

Install requirement (curl)

RUN apt-get update && apt-get install -y curl

Install

RUN curl -L -O https://artifacts.elastic.co/downloads/beats/metricbeat/metricbeat-7.5.1-amd64.deb &&
dpkg -i metricbeat-7.5.1-amd64.deb

COPY ./metricbeat.yml /usr/share/metricbeat/metricbeat.yml
COPY ./metricbeat.yml /etc/metricbeat/metricbeat.yml

COPY ./modules.d /etc/metricbeat/modules.d
RUN cd /usr/share/metricbeat/ &&
./bin/metricbeat modules enable mysql &&
service metricbeat start

EXPOSE 3306

[Martin_Kirouac]

I realize files are not placed where the exe was expecting them. So in /usr/share/metricbeat, there is a bin where the metricbeat exe is and the kibana directory. The exe is looking for that directory but at his level (./bin directory). So I created a symbolic link to the kibana directory.

Index setup finished.

Loading dashboards (Kibana must be running and reachable)

Exiting: Failed to import dashboard: Failed to load directory /usr/share/metricbeat/bin/kibana/7/dashboard:

error loading /usr/share/metricbeat/bin/kibana/7/dashboard/Metricbeat-aerospike-overview.json: index [.kibana_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];. Response: {"objects":[{"id":"b15668d0-bac3-11e9-a579-f5c0a5d81340","type":"dashboard","error":{"message":"index [.kibana_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];"}},{"id":"72523510-bac3-11e9-a579-f5c0a5d81340","type":"visualization",... (truncated)

[Martin_Kirouac]

Looks related to Kibana security as I am seeing this on the Kibana server :

{"type":"log","@timestamp":"2020-01-15T13:07:17Z","tags":["error","task_manager"],"pid":7,"message":"Failed to poll for work: Authorization Exception :: {"path":"/.kibana_task_manager/_update_by_query","query":{"ignore_unavailable":true,"refresh":true,"max_docs":10,"conflicts":"proceed"},"body":"{\"query\":{\"bool\":{\"must\":[{\"term\":{\"type\":\"task\"}},{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"bool\":{\"must\":[{\"term\":{\"task.status\":\"idle\"}},{\"range\":{\"task.runAt\":{\"lte\":\"now\"}}}]}},{\"bool\":{\"must\":[{\"bool\":{\"should\":[{\"term\":{\"task.status\":\"running\"}},{\"term\":{\"task.status\":\"claiming\"}}]}},{\"range\":{\"task.retryAt\":{\"lte\":\"now\"}}}]}}]}},{\"bool\":{\"should\":[{\"exists\":{\"field\":\"task.interval\"}},{\"bool\":{\"must\":[{\"term\":{\"task.taskType\":\"vis_telemetry\"}},{\"range\":{\"task.attempts\":{\"lt\":3}}}]}},{\"bool\":{\"must\":[{\"term\":{\"task.taskType\":\"lens_telemetry\"}},{\"range\":{\"task.attempts\":{\"lt\":3}}}]}}]}}]}}]}},\"sort\":{\"_script\":{\"type\":\"number\",\"order\":\"asc\",\"script\":{\"lang\":\"expression\",\"source\":\"doc['task.retryAt'].value || doc['task.runAt'].value\"}}},\"seq_no_primary_term\":true,\"script\":{\"source\":\"ctx._source.task.ownerId=params.ownerId; ctx._source.task.status=params.status; ctx._source.task.retryAt=params.retryAt;\",\"lang\":\"painless\",\"params\":{\"ownerId\":\"2bf8ba17-096e-45c2-8fb1-743c9c3df462\",\"retryAt\":\"2020-01-15T13:07:47.610Z\",\"status\":\"claiming\"}}}","statusCode":403,"response":"{\"took\":2,\"timed_out\":false,\"total\":2,\"updated\":0,\"deleted\":0,\"batches\":1,\"version_conflicts\":0,\"noops\":0,\"retries\":{\"bulk\":0,\"search\":0},\"throttled_millis\":0,\"requests_per_second\":-1.0,\"throttled_until_millis\":0,\"failures\":[{\"index\":\".kibana_task_manager_1\",\"type\":\"_doc\",\"id\":\"task:Lens-lens_telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\".kibana_task_manager_1\",\"type\":\"_doc\",\"id\":\"task:oss_telemetry-vis_telemetry\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [.kibana_task_manager_1] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}]}"}"}

[shaunak (Shaunak Kashyap)]

Hi @Martin_Kirouac,

Firstly, welcome to the Elastic community forums!

A couple of things about your setup:

1. Rather than creating a new Docker image based on the MySQL one and adding Metricbeat to it, have you considered running Metricbeat as a sidecar container to your MySQL container? You could then use Metricbeat's autodiscovery mechanism to tell Metricbeat to run with the mysql module whenever it discovers a MySQL container running in Docker.

2. As for Kibana dashboards setup, I see that you have set setup.dashboards.enabled: true and setup.kibana.* settings in your configuration. With these set, you don't need to manually run metricbeat setup. Metricbeat should automatically connect to the Kibana API upon start up and load the dashboards.

Shaunak

[Martin_Kirouac]

Hi Shaunak. Thanks for your response. I actually have the Metricbeat based image but I didn't know it could be use that way. I will definitely look it closer and switch. Most other logging tools I have used in the past needed to be install on the server itself so I was assuming that by default.

Concerning point 2. Good to know. I am so new to Metricbeat that I was just "lighting up" everything for now in hope it starts :-D.

Thank you for your help.

[Martin_Kirouac]

So. I loaded the Metricbeat docker but I get he same error about the kibana directory path even in the Elastic Metricbeat image.

Exiting: Error importing Kibana dashboards: fail to import the dashboards in Kibana: Error importing directory /usr/share/metricbeat/kibana: Failed to import dashboard: Failed to load directory /usr/share/metricbeat/kibana/7/dashboard:

metricbeat:
image: elastic/metricbeat:7.5.1
container_name: metricbeat
ports:
- 5066
depends_on:
- 'kibana'
- 'elasticsearch'
volumes:
- ./docker/metricbeat/config/metricbeat.yml:/usr/share/metricbeat/metricbeat.yml
- ./docker/metricbeat/config/modules.d:/etc/metricbeat/modules.d