Filebeat filestream reads the files from last line after service restart

Elastic Stack Beats
1

how can i setup filebeat so after each filebeat service restart it start reading the files from last line and stop it from reading the file from beginning.

i have a config like the following:

filebeat.inputs:
- type: filestream
  id: my-filestream-id
  paths:
    - /var/log/messages
    - /var/log/*.log

can i set filebeat.registry.path: /dev/null ? is there away to stop filebeat from reading old logs?

2

Hi Jack_a, Filebeat, by default, reads new logs (except those that have already been sent to Elasticsearch)

From the doc -

Filebeat keeps the state of each file and frequently flushes the state to disk in the registry file. The state is used to remember the last offset a harvester was reading from and to ensure all log lines are sent.

3

hi, thanks for the reply. what i want is the filebeat start reading from last line of a log file when it stars. assume i have a file that have the logs for the past 5 years and still being updated. i want when i start logstash it starts reading new logs (since filebeat start) and not start from beginning of the file and reading logs from 5 years ago.

4
  1. The best approach is rotate the file and create a new one for new logs. Set ignore_older to not ingest previous file logs. Though if you can archive or move to different path, then you don't need to set ignore_older.
  2. You can give a try by setting exclude_lines. I haven't try but you can add regex on timestamp in your logs (specifically date or year) but you need to handle properly.
5

so i gues there is no direct approch. for example like what we can do with logstash and configure it like below

sincedb_path: /dev/null   # Ignore read state on restart
start_position: end        # Read from the end of the log file