Logstash: per-pipeline logfiles

I have a Logstash development environment with multiple pipelines. From time to time, I accidentally introduce an error into the config. In such a case, I would find it very useful to have a per-pipeline logfile, in addition to logstash-plain.log.

If my pipelines are called A, B, and C, I would like to have a logfile for each pipeline containing only events which affect that pipeline (e.g. logstash-plain.pipeline.A.log, logstash-plain.pipeline.B.log, logstash-plain.pipeline.C.log etc. or something along those lines).

Particularly when debug logging is enabled, the central log file is often very large and can be difficult to work with, even using grep or similar to filter out less-interesting lines. My hope is that by monitoring per-pipeline logs, it would be easy to pick out the affected pipeline(s).

For example, consider this log line:

    [2019-08-15T15:43:29,989][WARN ][org.logstash.plugins.pipeline.PipelineBus]
Attempted to send event to '%{[@metadata][pipeline][next]}' but that address was unavailable.
Maybe the destination pipeline is down or stopping? Will Retry.

Based on that line alone, I don't know which pipelines are involved, and I hope that per-pipeline logs would help with that.

The concept of per-pipeline logs was raised last year by @CasMeiron, but unfortunately the topic didn't receive a response. Is there a config option I've overlooked, or would this functionality require an enhancement?

Alternatively, is there a better way of thinking about this problem?

We’re working on this.
We have a Github issue and at the moment we only add the pipeline.id on each log line IF the log is related to a specific pipeline.

We have another Pull Request (visible here) where we plan to have one log file per pipeline.

Both of those features are not released, but you can follow the development in those issues.

As this is a core Logstash feature, it will be necessary to update to new versions of Logstash.

Hope this will help you in the future!


Thank you @Luca_Belluccini. Glad that this feature is already under development.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

The issue https://github.com/elastic/logstash/issues/11074 is closed.

It is now possible to have separate logs using pipeline.separate_logs: true in logstash.yml (starting 7.5.0).