Rolling pipelines.separate_logs

[chris_ts24]

Logstash v.7.13.2

Hi all,

I'm successfully running 3 logstash instances via the following pipelines.yml file:

- pipeline.id: upsert
  queue.type: persisted
  queue.checkpoint.writes: 1
  path.config: "<PATH TO UPSERT CONFIG>"
  path.queue: "<PATH TO UPSERT QUEUE>"
- pipeline.id: delete
  queue.type: persisted
  queue.checkpoint.writes: 1
  path.config: "<PATH TO DELETE CONFIG>"
  path.queue: "<PATH TO DELETE QUEUE>"
- pipeline.id: winlogs
  path.config: "<PATH TO WINLOGS CONFIG>"
  path.queue: "<PATH TO WINLOGS QUEUE>"

My logstash.yml file:

path.data: "/home/ts24/apps/logstash/data"
path.logs: "/home/ts24/apps/logstash/logs"
pipeline.unsafe_shutdown: true
pipeline.separate_logs: true

The problem I face is follows: After a day of logging, none of the individual pipeline's logs are rolled over.

The rolled logs displayed in the in the picture (those with the timestamps) contain information that the single pipelines logstash instance itself outputs, but each individual pipeline log (upsert, delete, winlogs) does not get rolled.

Is there a way to define log4j2 properties for each individual pipeline? Or perhaps a way to ensure rolling for every individual pipeline? Currently, I have defined those parameters within a log4j2.properties file that can be found within the defined in my pipelines.yml file, but this doesn't seem to be functioning correctly. Otherwise, I have the default log4j2.properties file in the same location as my pipelines.yml file.

Happy for any help / advice!

[Badger]

@chris_ts24

Is there a way to define log4j2 properties for each individual pipeline? Or perhaps a way to ensure rolling for every individual pipeline?

I am certainly no log4j2 expert but looking at the log4j2.properties I think per-pipeline logs are only rolled based on size, and not time.

You could try reducing appender.routing.pipeline.policy.size to see if that causes them to roll. If it does you could try defining a TimeBasedTriggeringPolicy in the appender.routing.pipeline section and may well find out why that was not done

[chris_ts24]

Thanks again @Badger. I've added a TimeBasedTriggeringPolicy to our routing pipeline. Let's see what happens by tomorrow and see if I found out why we shouldn't do this

[chris_ts24]

So I have a log of about 1GB now. I guess TimeBasedTriggering is not something I'll be doing in this case.