Monitoring ingress pipelines

version 7.14

I have just started trying to use an ingest pipeline on some indices produced by winlogbeats. There are two indices because there are two different versions of the agent.

I have a very simple pipeline with a single drop that matches event.code == 3 (network connections). I have tested it in Kibana against documents from both indices and it works as expected.

The first index I applied it to worked fine but I when applied it to the second index and nothing was indexed at all! Worse still when I removed the setting from the index there were no documents added to the index.

I have confirmed ( with tcpdump) that data is still being sent to that index and (usual provisio) "nothing else has changed"™

I have tried to force the index to rollover without success -- I assume that the rover check is done when data is added to the index.

Is there anyway that I can monitor what the ingest pipeline is doing?

Any other thoughts on diagnosing what is going on.

I think I have finally figured out what the problem was. Very simple as always -- I had failed to add the "ingest" role to one of the ES servers.

It really is a unhelpful failure mode. The really puzzling thing was that as soon as I removed the pipeline clause (after several hours of no logs being indexed) they "magically" appeared. In hindsight winlogbeats must have go failures and stashed the logs.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.