Kibana dashboard cannot load metricbeat data

[leelax22 (changmin LEE)]

I am testing to monitor aks using ELK stack and beat.

The installation is completed with version 8.1.0, and it is confirmed that data is well accumulated in Kibana through the metricbeat-* index.

Because I did not connect directly from metricbeat to Elasticsearch and went through logstash, it was necessary to install a dashboard separately, and it was also completed by connecting to the metricbeat pod separately.

./metricbeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.enabled=true \
  -E output.elasticsearch.hosts=['https://elasticsearch-master:9200'] \
  -E output.elasticsearch.ssl.verification_mode=none \
  -E output.elasticsearch.username=elastic \
  -E output.elasticsearch.password=******* \
  -E setup.kibana.ssl.enabled=true \
  -E setup.kibana.host="https://kibana-kibana:5601" \
  -E setup.kibana.ssl.verification_mode=none \
  -E setup.ilm.overwrite=true

First, I have a total of 3 nodes, but there is a problem in the log that [kubernetes.node.name] is split.

Second, despite the fact that there are logs that are clearly collected as [kubernetes.node.name],

Nodes [Metricbeat Kubernetes] ECS in visualize library fails to load those items. Same problem with pods and deployments.

Where should I look? If anyone knows, please help.

[leelax22 (changmin LEE)]

first problem solved.
I use [kubernetes.node.name.keyword] instead [kubernetes.node.name]
it works.

[stephenb (Stephen Brown)]

The reason you are having these problems is that you need to run complete metricbeat setup not just load the dashboards

metricbeat setup also loads. The templates for the mappings are correct. The fact that you're having to use keywords means the mapping is not correct.

Take a look at this post here

[leelax22 (changmin LEE)]

Thank you very much for your reply. The process seems a bit cumbersome. I will try it and leave a review.

[stephenb (Stephen Brown)]

It's not my process. It's how it works.

If you want to use metricbeat correctly, you need to run setup, and set up all the assets, not just dashboards.

After you do that, if you want to pump it through logstash, that's fine

In short, I always say run metricbeat directly to Elasticsearch and Once that all works then send metricbeat data through logstash.

At this point, logstash is just a collecting forward mechanism.

Pretty straight forward.