# Logstash log4j2.properties configuration for pipeline logs

**URL:** <https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-pipeline-logs/278612>\
**Category:** Logstash\
**Tags:** elastic-stack-monitoring\
**Created:** [July 14, 2021, 6:02am UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-pipeline-logs/278612 "2021-07-14T06:02:20Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Kshema](https://avatars.discourse-cdn.com/v4/letter/k/cc9497/32.png) [@Kshema](https://discuss.elastic.co/u/Kshema)\
**Post date:** [July 14, 2021, 6:02am UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-pipeline-logs/278612/1 "2021-07-14T06:02:20Z")

</div>

In our application, `logstash.yml` is configured to produce separate log files for each pipeline with the help of below property  
`pipeline.separate_logs: true`

This configuration setup is creating the below mentioned logs:

```auto
logstash-plain.log
pipeline_<pipeline_id>.log

```

`logstash_plain.log` is getting converted to a zip file `logstash-plain-<date>.log.gz` each day but `piepline_<pipeline_id>.log` is observed to be getting appended to the same file for every logstash run. Pipeline log turned out to be occupying huge space now.

Looking for a way to configure `log4j2.properties` file so that pipeline log ( `pipeline_<pipeline_id>.log` ) can also be transformed to a zip file after a due limit is reached (SizeBased or TimeBased) just like the plain log.

Appreciate any help on this. Thanks

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [July 14, 2021, 4:06pm UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-pipeline-logs/278612/2 "2021-07-14T16:06:15Z")

</div>

In the log4j2.properties file you need to adjust the following settings:

```auto
appender.routing.routes.route_pipelines.rolling.policy.size = 100MB
appender.routing.routes.route_pipelines.strategy.type = DefaultRolloverStrategy
appender.routing.routes.route_pipelines.strategy.max = 30

```

This means that at every 100MB do a rollover of a log and keep 30 of those rollover log files.  
Adjust these numbers to best suit your environment.

---

<div class="post-metadata">

**Author:** ![Kshema](https://avatars.discourse-cdn.com/v4/letter/k/cc9497/32.png) [@Kshema](https://discuss.elastic.co/u/Kshema)\
**Post date:** [July 16, 2021, 2:16pm UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-pipeline-logs/278612/3 "2021-07-16T14:16:31Z")

</div>

Thanks, @AquaX

The size based roll over configuration was working for me. However, the removal of rollover logs is not working as configured .

Can you please suggest on why the removal was not working based on the below configurations

```auto
appender.routing.routes.route_pipelines.rolling.policy.type = SizeBasedTriggeringPolicy
appender.routing.routes.route_pipelines.rolling.policy.size = 2KB
appender.routing.routes.route_pipelines.strategy.max = 2

```

I can see upto 10 rolled over logs after I ran logstash multiple times.

Also, please suggest how pipeline logs( `piepline_<pipeline_id>.log` ) can be rolled over based on _TimeBasedTriggeringPolicy_

---

<div class="post-metadata">

**Author:** ![mruthyu](https://avatars.discourse-cdn.com/v4/letter/m/bb73d2/32.png) [@mruthyu](https://discuss.elastic.co/u/mruthyu)\
**Post date:** [July 19, 2021, 11:11am UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-pipeline-logs/278612/4 "2021-07-19T11:11:10Z")

</div>

I also have a similar requirement. Any inputs on this issue will be helpful. Removal of logs will help us to save the disc space.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 16, 2021, 11:11am UTC](https://discuss.elastic.co/t/logstash-log4j2-properties-configuration-for-pipeline-logs/278612/5 "2021-08-16T11:11:49Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.