Monitoring ingress pipelines

[Russell_Fulton]

version 7.14

I have just started trying to use an ingest pipeline on some indices produced by winlogbeats. There are two indices because there are two different versions of the agent.

I have a very simple pipeline with a single drop that matches event.code == 3 (network connections). I have tested it in Kibana against documents from both indices and it works as expected.

The first index I applied it to worked fine but I when applied it to the second index and nothing was indexed at all! Worse still when I removed the setting from the index there were no documents added to the index.

I have confirmed ( with tcpdump) that data is still being sent to that index and (usual provisio) "nothing else has changed"™

I have tried to force the index to rollover without success -- I assume that the rover check is done when data is added to the index.

Is there anyway that I can monitor what the ingest pipeline is doing?

Any other thoughts on diagnosing what is going on.

[Russell_Fulton]

I think I have finally figured out what the problem was. Very simple as always -- I had failed to add the "ingest" role to one of the ES servers.

It really is a unhelpful failure mode. The really puzzling thing was that as soon as I removed the pipeline clause (after several hours of no logs being indexed) they "magically" appeared. In hindsight winlogbeats must have go failures and stashed the logs.