I'm successfully running 3 logstash instances via the following pipelines.yml file:
- pipeline.id: upsert queue.type: persisted queue.checkpoint.writes: 1 path.config: "<PATH TO UPSERT CONFIG>" path.queue: "<PATH TO UPSERT QUEUE>" - pipeline.id: delete queue.type: persisted queue.checkpoint.writes: 1 path.config: "<PATH TO DELETE CONFIG>" path.queue: "<PATH TO DELETE QUEUE>" - pipeline.id: winlogs path.config: "<PATH TO WINLOGS CONFIG>" path.queue: "<PATH TO WINLOGS QUEUE>"
My logstash.yml file:
path.data: "/home/ts24/apps/logstash/data" path.logs: "/home/ts24/apps/logstash/logs" pipeline.unsafe_shutdown: true pipeline.separate_logs: true
The problem I face is follows: After a day of logging, none of the individual pipeline's logs are rolled over.
The rolled logs displayed in the in the picture (those with the timestamps) contain information that the single pipelines logstash instance itself outputs, but each individual pipeline log (upsert, delete, winlogs) does not get rolled.
Is there a way to define log4j2 properties for each individual pipeline? Or perhaps a way to ensure rolling for every individual pipeline? Currently, I have defined those parameters within a log4j2.properties file that can be found within the defined in my pipelines.yml file, but this doesn't seem to be functioning correctly. Otherwise, I have the default log4j2.properties file in the same location as my pipelines.yml file.
Happy for any help / advice!