In my setup I have 3 Logstash(7.5.0) nodes, each node has 20 pipelines. I'm trying to troubleshoot some performance issues with one of my pipelines, but find it difficult because the data is not always viewable in Kibana. (I have a few GROK expressions that are taking a long time to complete).
I'm not sure the best way to troubleshoot this, or get to a resolution. My understanding is that every 10 seconds or so Logstash posts data into ES. Once that data is in ES it should be viewable in Kibana (X Pack is enabled).
From my experience It kind of seems like its more its more likely to to appear if Logstash has been running for some time, but sometimes it appears quite quickly after a restart.(I think). When this is happening I can see the event rate for the logstash nodes so I know it is reporting some monitoring data to ES, but not all.
In Kibana I try multiple things, like clicking on the pipeline tab from the context of all logstash nodes, or one logstash node.. setting the time to 1 minute or 1 hour.. I just can not consistently get the information to appear.