Logstash log4j2.properties configuration for pipeline logs

[Kshema]

In our application, logstash.yml is configured to produce separate log files for each pipeline with the help of below property
pipeline.separate_logs: true

This configuration setup is creating the below mentioned logs:

logstash-plain.log
pipeline_<pipeline_id>.log

logstash_plain.log is getting converted to a zip file logstash-plain-<date>.log.gz each day but piepline_<pipeline_id>.log is observed to be getting appended to the same file for every logstash run. Pipeline log turned out to be occupying huge space now.

Looking for a way to configure log4j2.properties file so that pipeline log ( pipeline_<pipeline_id>.log ) can also be transformed to a zip file after a due limit is reached (SizeBased or TimeBased) just like the plain log.

Appreciate any help on this. Thanks

[AquaX (Andreas Helmer)]

In the log4j2.properties file you need to adjust the following settings:

appender.routing.routes.route_pipelines.rolling.policy.size = 100MB
appender.routing.routes.route_pipelines.strategy.type = DefaultRolloverStrategy
appender.routing.routes.route_pipelines.strategy.max = 30

This means that at every 100MB do a rollover of a log and keep 30 of those rollover log files.
Adjust these numbers to best suit your environment.

[Kshema]

Thanks, @AquaX

The size based roll over configuration was working for me. However, the removal of rollover logs is not working as configured .

Can you please suggest on why the removal was not working based on the below configurations

appender.routing.routes.route_pipelines.rolling.policy.type = SizeBasedTriggeringPolicy
appender.routing.routes.route_pipelines.rolling.policy.size = 2KB
appender.routing.routes.route_pipelines.strategy.max = 2

I can see upto 10 rolled over logs after I ran logstash multiple times.

Also, please suggest how pipeline logs( piepline_<pipeline_id>.log ) can be rolled over based on TimeBasedTriggeringPolicy

[mruthyu (MJay)]

I also have a similar requirement. Any inputs on this issue will be helpful. Removal of logs will help us to save the disc space.