Filebeat does not harvest files after a restart

[Amit_Periyapatna]

I have configured filebeat to read files from a log directory.

Any time I change a configuration in filebeat.yml file and restart filebeat, it stops harvesting files.

If I restart after deleting the registry, everything works fine. I am not sure if it is a bug within filebeat that it requires me to delete the registry every time I change something in the configuration.

Use case:
I'm trying to add a new log directory with a new field in my filebeat configuration as follows:

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - "/mylogs/something*"
    - "/mylogs/otherthing*"

  ignore_older: 5m
  scan_frequency: 20s
  fields:
    type: other
  fields_under_root: true
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:|^[[:space:]]{2}\"|^}'
  multiline.negate: false
  multiline.match: after
# This is the configuration that I added to filebeat before restarting it.
- type: log
  enabled: true
  paths:
    - "/mylogs/newtype*"
  ignore_older: 5m
  fields:
    type: newtype
  fields_under_root: true
  multiline.pattern: '^[[:space:]]+(at|\.{3})\b|^Caused by:|^[[:space:]]{2}\"|^}'
  multiline.negate: false
  multiline.match: after

Let me know if any more details are required.

[steffens (Steffen Siering)]

Filebeat remembers the last processed offsets in a file. It does not send already processed contents when the configuration changed.

[Amit_Periyapatna]

Thank you for your reply. I agree that filebeat does that, but as soon as I change anything on my configuration file and restart filebeat, it does not send any data to logstash unless I delete the registry. I am not sure if I am doing something wrong but it has been happening for the last couple of weeks.

[steffens (Steffen Siering)]

What kind of changes have you done? Can you share the exact config files before and after change?

When filebeat doesn't send it would be helpful to have a look at debug logs from filebeat.