Metricbeat + Istio 1.26: Missing Data on Dashboard and No Fields in Discover

Elastic Stack Beats
1

Hi all,

I'm currently setting up Metricbeat to collect telemetry data from Istio 1.26 running in a Kubernetes cluster, and I'm encountering issues where the default dashboards show no data, and the expected Istio fields are missing in Kibana Discover.

WhatsApp Image 2025-07-14 at 09.58.32_a3bdff13
WhatsApp Image 2025-07-14 at 09.58.32_a3bdff131600×841 141 KB

WhatsApp Image 2025-07-14 at 09.58.14_1d60f563
WhatsApp Image 2025-07-14 at 09.58.14_1d60f5631600×852 192 KB

:pushpin: Setup Summary

  • Metricbeat version: 9.0.3
  • Istio version: 1.26
  • Deployment method: Metricbeat deployed as DaemonSet in Kubernetes
  • Output: Sending directly to Elasticsearch (https://10.130.234.70:9200)
  • Kibana host: http://10.130.234.72:5601

:gear: Metricbeat Setup Commands Used

1. To load index template:

metricbeat setup --index-management \
  -E output.logstash.enabled=false \
  -E 'output.elasticsearch.hosts=["https://10.130.234.70:9200"]' \
  -E 'output.elasticsearch.username=username' \
  -E 'output.elasticsearch.password=password' \
  -E 'output.elasticsearch.ssl.certificate_authorities=["/elastic/ca.crt"]'

2. To load dashboards:

metricbeat setup -e \
  -E output.logstash.enabled=false \
  -E output.elasticsearch.hosts=['localhost:9200'] \
  -E output.elasticsearch.username=metricbeat_internal \
  -E output.elasticsearch.password=YOUR_PASSWORD \
  -E setup.kibana.host=localhost:5601

:red_exclamation_mark: Current Issue

  • The OOTB Istio dashboards are showing empty charts/widgets.
  • In Kibana Discover, when I filter metricset.name: proxy or metricset.name: mesh, there are no results.
  • Expected fields such as istio.request.count, istio.request.duration, etc. do not exist in the index pattern.

:red_question_mark:What I Need Help With

  1. What else should I check to make sure Metricbeat is collecting the data correctly?
  2. Are there additional steps needed to get Istio 1.26 metrics to appear in Kibana?
  3. How do I verify whether Metricbeat is failing at the collection stage vs. indexing stage?

Any pointers or help debugging would be greatly appreciated!

Thanks in advance!

2

Hello @Kevin_Rizki

Welcome back!!

Could you please confirm if you enabled the istio module?

metricbeat modules enable istio

Once you enable you can see the details in modules.d/istio.yml

You can edit the yml as per the data you want to collect , below is the reference link :

If you have already followed these steps than maybe can review the metrcibeat logs to understand why the istio logs are not sent to the elasticsearch node.

Thanks!!

3

Thanks. So far we already test the module enable it looks like it turns on. Some of the Istio data it actually receive on Elasticsearch.

Here I show last log coming from the metricbeat pods:

{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.75:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.74:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.76:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.76:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.74:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.73:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.75:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.74:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.76:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.75:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.845Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":138},"message":"Connecting to backoff(async(tcp://10.130.234.73:6021))","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.74:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.76:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.75:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.76:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.74:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.75:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.73:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.76:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.74:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.75:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:36.846Z","log.logger":"publisher_pipeline_output","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run","file.name":"pipeline/client_worker.go","file.line":146},"message":"Connection to backoff(async(tcp://10.130.234.73:6021)) established","service.name":"metricbeat","ecs.version":"1.6.0"}
{"log.level":"info","@timestamp":"2025-07-14T03:01:46.790Z","log.logger":"monitoring","log.origin":{"function":"github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).logSnapshot","file.name":"log/log.go","file.line":192},"message":"Non-zero metrics in the last 30s","service.name":"metricbeat","monitoring":{"metrics":{"beat":{"cgroup":{"memory":{"mem":{"usage":{"bytes":0}}}},"cpu":{"system":{"ticks":119040,"time":{"ms":20}},"total":{"ticks":1958760,"time":{"ms":210},"value":1958760},"user":{"ticks":1839720,"time":{"ms":190}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":27},"info":{"ephemeral_id":"b38c8ccc-a817-4aec-9744-bc3fe0d1eaff","uptime":{"ms":233910062},"version":"9.0.3"},"memstats":{"gc_next":67829226,"memory_alloc":49876992,"memory_total":549959253376,"rss":213151744},"runtime":{"goroutines":145}},"libbeat":{"config":{"module":{"running":1},"scans":3},"output":{"events":{"acked":2503,"active":0,"batches":3,"total":2503},"read":{"bytes":24},"write":{"bytes":301179,"latency":{"histogram":{"count":42225,"max":2000,"mean":48.3662109375,"median":30,"min":4,"p75":66,"p95":139,"p99":190.5,"p999":1958.0500000000382,"stddev":74.43991174001857}}}},"pipeline":{"clients":3,"events":{"active":284,"published":2202,"retry":6666,"total":2202},"queue":{"acked":2503,"added":{"events":2202},"consumed":{"events":2503},"filled":{"bytes":0,"events":284,"pct":0.08875},"max_bytes":0,"max_events":3200,"removed":{"events":2503}}}},"metricbeat":{"istio":{"istiod":{"events":147,"success":147},"proxy":{"events":63,"success":63}},"prometheus":{"collector":{"events":1992,"success":1992}}},"system":{"load":{"1":0.89,"15":0.32,"5":0.43,"norm":{"1":0.1113,"15":0.04,"5":0.0538}}}},"ecs.version":"1.6.0"}}