Configure filebeat to control how often logs are read and to ignore old logs

Elastic Stack Beats
1

Hi,
I have a setup in my filebeat.yml roughly as follows:

filebeat.propectors:
- type: log
paths:
- /tmp/log/typeA*.log
pipeline: "pipelineA"
fields_under_root: true
fields:
logtype: TYPEA
- type: log
paths:
- /tmp/log/typeB*.log
pipeline: "pipelineB"
fields_under_root: true
fields:
logtype: TYPEB
- type: log
paths:
- /tmp/log/typeC*.log
pipeline: "pipelineC"
fields_under_root: true
fields:
logtype: TYPEC

It works fine but I need control on reading. I would like to be able to control the input so that some logs are read more frequently than (e.g. for pipelineC, read only once a day, but for pipelineA, read every minute). I would also like to be able to ignore logs older than a certain age (e.g. ignore logs older than 3 days).

I would be grateful for any ideas. Thank you

2

I will answer this myself:

filebeat.propectors:

  • type: log
    paths:
  • /tmp/log/typeA*.log
    ignore_older: 72h
    scan_frequency: 10s
    pipeline: "pipelineA"
    fields_under_root: true
    fields:
    logtype: TYPEA
  • type: log
    paths:
  • /tmp/log/typeB*.log
    ignore_older: 72h
    scan_frequency: 10s
    pipeline: "pipelineB"
    fields_under_root: true
    fields:
    logtype: TYPEB
  • type: log
    paths:
  • /tmp/log/typeC*.log
    ignore_older: 72h
    scan_frequency: 10s
    pipeline: "pipelineC"
    fields_under_root: true
    fields:
    logtype: TYPEC
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.