I have a relatively complicated logstash pipeline setup, with some pipelines feeding into others, splitting events, making http calls to external services, and sometimes feeding an event back into the pipeline it came from. (There is logic to prevent an infinite loop).
I'm trying to write some integration tests that feed test event(s) into a running logstash, wait for logstash to finish processing them completely (including any extra events they spawned), then check that the resulting output is as expected.
This logstash instance shouldn't be receiving any additional input from elsewhere, so I think it would be sufficient to check that it was "idle" (ignoring any events to do with the xpack monitoring).
I think the pipleine stats monitoring API is probably what I want to use - https://www.elastic.co/guide/en/logstash/current/node-stats-api.html#pipeline-stats - but I'm unsure. If the values for "in" and "out" for every pipeline are equal, does that mean I can be certain that there's nothing more "in flight"? Or is it possible for these counters to be out of sync for some other reason e.g. event splitting, event filtering?
Discussions at Pipeline stats API: in/out/filtered (and following the links) appear to suggest that "in" and "out" will always appear in sync, with a bug suggesting this shouldn't be the case - https://github.com/elastic/logstash/issues/8752 - and a related bug shows a situation where in and out are different - https://github.com/elastic/logstash/issues/8753