Logstash log4j2.properties configuration for pipeline logs

In our application, logstash.yml is configured to produce separate log files for each pipeline with the help of below property
pipeline.separate_logs: true

This configuration setup is creating the below mentioned logs:


logstash_plain.log is getting converted to a zip file logstash-plain-<date>.log.gz each day but piepline_<pipeline_id>.log is observed to be getting appended to the same file for every logstash run. Pipeline log turned out to be occupying huge space now.

Looking for a way to configure log4j2.properties file so that pipeline log ( pipeline_<pipeline_id>.log ) can also be transformed to a zip file after a due limit is reached (SizeBased or TimeBased) just like the plain log.

Appreciate any help on this. Thanks

In the log4j2.properties file you need to adjust the following settings:

appender.routing.routes.route_pipelines.rolling.policy.size = 100MB
appender.routing.routes.route_pipelines.strategy.type = DefaultRolloverStrategy
appender.routing.routes.route_pipelines.strategy.max = 30

This means that at every 100MB do a rollover of a log and keep 30 of those rollover log files.
Adjust these numbers to best suit your environment.

Thanks, @AquaX

The size based roll over configuration was working for me. However, the removal of rollover logs is not working as configured .

Can you please suggest on why the removal was not working based on the below configurations

appender.routing.routes.route_pipelines.rolling.policy.type = SizeBasedTriggeringPolicy
appender.routing.routes.route_pipelines.rolling.policy.size = 2KB
appender.routing.routes.route_pipelines.strategy.max = 2

I can see upto 10 rolled over logs after I ran logstash multiple times.

Also, please suggest how pipeline logs( piepline_<pipeline_id>.log ) can be rolled over based on TimeBasedTriggeringPolicy

I also have a similar requirement. Any inputs on this issue will be helpful. Removal of logs will help us to save the disc space.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.