Logstash Pipeline Monitoring Data Is Intermittently Blank In Kibana

[scott_stash]

In my setup I have 3 Logstash(7.5.0) nodes, each node has 20 pipelines. I'm trying to troubleshoot some performance issues with one of my pipelines, but find it difficult because the data is not always viewable in Kibana. (I have a few GROK expressions that are taking a long time to complete).

I'm not sure the best way to troubleshoot this, or get to a resolution. My understanding is that every 10 seconds or so Logstash posts data into ES. Once that data is in ES it should be viewable in Kibana (X Pack is enabled).

From my experience It kind of seems like its more its more likely to to appear if Logstash has been running for some time, but sometimes it appears quite quickly after a restart.(I think). When this is happening I can see the event rate for the logstash nodes so I know it is reporting some monitoring data to ES, but not all.

In Kibana I try multiple things, like clicking on the pipeline tab from the context of all logstash nodes, or one logstash node.. setting the time to 1 minute or 1 hour.. I just can not consistently get the information to appear.

[tomx1]

Same issue here... It's a kind of frustrating to debug "Stack Monitoring" issues because there are not meaningful logfiles.
The feature overall is instable in my opinion... Since I've upgraded to the newest version of ELK, stack monitoring does not work at all if collecting monitoring data with metricbeat shipped to a dedicated monitoring cluster.

[scott_stash]

The one thing that is interesting is that if I make changes to my pipeline and restart Kibana shows the updated pipeline almost immediately, right now I added another GROK expression and it shows up in the monitoring section right away, but the stats are empty.

The more I test it it seems like it might just take 20-30 minutes after a logstash restart to show up.