# Rolling pipelines.separate\_logs

**URL:** <https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925>\
**Category:** Logstash\
**Created:** [July 6, 2021, 8:10am UTC](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925 "2021-07-06T08:10:36Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![chris\_ts24](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_ts24/32/79048_2.png) [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\
**Post date:** [July 6, 2021, 8:10am UTC](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925/1 "2021-07-06T08:10:36Z")

</div>

Logstash v.7.13.2

Hi all,

I'm successfully running 3 logstash instances via the following pipelines.yml file:

```auto
- pipeline.id: upsert
  queue.type: persisted
  queue.checkpoint.writes: 1
  path.config: "<PATH TO UPSERT CONFIG>"
  path.queue: "<PATH TO UPSERT QUEUE>"
- pipeline.id: delete
  queue.type: persisted
  queue.checkpoint.writes: 1
  path.config: "<PATH TO DELETE CONFIG>"
  path.queue: "<PATH TO DELETE QUEUE>"
- pipeline.id: winlogs
  path.config: "<PATH TO WINLOGS CONFIG>"
  path.queue: "<PATH TO WINLOGS QUEUE>"

```

My logstash.yml file:

```auto
path.data: "/home/ts24/apps/logstash/data"
path.logs: "/home/ts24/apps/logstash/logs"
pipeline.unsafe_shutdown: true
pipeline.separate_logs: true

```

The problem I face is follows: After a day of logging, none of the individual pipeline's logs are rolled over.  
 ![pipeline-logs](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bb676b18248d29a4cf3503b58571f0633ffdcbff.png)  
The rolled logs displayed in the in the picture (those with the timestamps) contain information that the single **pipelines** logstash instance itself outputs, but each **individual pipeline** log (upsert, delete, winlogs) does not get rolled.

Is there a way to define log4j2 properties for each individual pipeline? Or perhaps a way to ensure rolling for every individual pipeline? Currently, I have defined those parameters within a log4j2.properties file that can be found within the defined in my pipelines.yml file, but this doesn't seem to be functioning correctly. Otherwise, I have the default log4j2.properties file in the same location as my pipelines.yml file.

Happy for any help / advice!

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [July 6, 2021, 4:14pm UTC](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925/2 "2021-07-06T16:14:48Z")

</div>

> [@chris\_ts24](#):
>
> Is there a way to define log4j2 properties for each individual pipeline? Or perhaps a way to ensure rolling for every individual pipeline?

I am certainly no log4j2 expert but looking at the log4j2.properties I think per-pipeline logs are only rolled based on size, and not time.

You could try reducing appender.routing.pipeline.policy.size to see if that causes them to roll. If it does you could try defining a TimeBasedTriggeringPolicy in the appender.routing.pipeline section and may well find out why that was not done 😃

---

<div class="post-metadata">

**Author:** ![chris\_ts24](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_ts24/32/79048_2.png) [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\
**Post date:** [July 9, 2021, 1:59pm UTC](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925/3 "2021-07-09T13:59:47Z")

</div>

Thanks again @Badger. I've added a TimeBasedTriggeringPolicy to our routing pipeline. Let's see what happens by tomorrow and see if I found out why we shouldn't do this 😅

---

<div class="post-metadata">

**Author:** ![chris\_ts24](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/chris_ts24/32/79048_2.png) [@chris\_ts24](https://discuss.elastic.co/u/chris_ts24)\
**Post date:** [July 12, 2021, 7:20am UTC](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925/4 "2021-07-12T07:20:06Z")

</div>

So I have a log of about 1GB now. 🤦‍♂️ I guess TimeBasedTriggering is not something I'll be doing in this case.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 9, 2021, 7:20am UTC](https://discuss.elastic.co/t/rolling-pipelines-separate-logs/277925/5 "2021-08-09T07:20:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.