I have a Logstash development environment with multiple pipelines. From time to time, I accidentally introduce an error into the config. In such a case, I would find it very useful to have a per-pipeline logfile, in addition to logstash-plain.log.
If my pipelines are called A, B, and C, I would like to have a logfile for each pipeline containing only events which affect that pipeline (e.g. logstash-plain.pipeline.A.log, logstash-plain.pipeline.B.log, logstash-plain.pipeline.C.log etc. or something along those lines).
Particularly when debug logging is enabled, the central log file is often very large and can be difficult to work with, even using grep or similar to filter out less-interesting lines. My hope is that by monitoring per-pipeline logs, it would be easy to pick out the affected pipeline(s).
For example, consider this log line:
[2019-08-15T15:43:29,989][WARN ][org.logstash.plugins.pipeline.PipelineBus]
Attempted to send event to '%{[@metadata][pipeline][next]}' but that address was unavailable.
Maybe the destination pipeline is down or stopping? Will Retry.
Based on that line alone, I don't know which pipelines are involved, and I hope that per-pipeline logs would help with that.
The concept of per-pipeline logs was raised last year by @CasMeiron, but unfortunately the topic didn't receive a response. Is there a config option I've overlooked, or would this functionality require an enhancement?
Alternatively, is there a better way of thinking about this problem?