Filebeat from type "log" to "filestream" duplicated events

Elastic Stack Beats
1

Hello,

we are changing from the filebeat type "log" to "filesteam"
Since we are changed, we have duplicates events in our logstash. Looks like this was the reason of my other problem with the logstash throttle and max_age time. Because the old log events are pushed some days later again.

We are not changing the default file_identity settings

Old config:

- type: log
  paths:
    - /var/log/commserverSpool.log
  fields:
   type: commserver_spool
   application: '${APPLICATION}'
   apptype: dc
  fields_under_root: true
  symlinks: true

New config:

- type: filestream
  id: '${FBHOST}:commserver_spool'
  paths:
    - /var/log/commserverSpool.log
  fields:
   type: commserver_spool
   application: '${APPLICATION}'
   apptype: dc
  fields_under_root: true
  symlinks: true

Any idea what I make wrong?

Thx
Thomas

2

Which version are you using?

The filestream input will work as a new input, so it will per default read the files from the start as it store the registry in a different way, so one of the issues of just changing log to filestream is that it can lead to duplication.

There is however an new option named take_over to avoid this, not sure in which version this was introduced, but to migrate from a log input to a filestream input you need to follow the steps described here: migrate from log to filestream.

3

Thank you for your response

We use this filebeat version:

filebeat version
filebeat version 8.14.3 (amd64), libbeat 8.14.3 [71819961045386b23edc18455f1b54764292816c built 2024-07-08 22:05:44 +0000 UTC]

The server is a new installed RHEL9 server. All log files are new. Is not really a migration, only the config. We are starting direct with the new filestream config. But the logs are repeating. The file is not rotating.

Now I make a rollback to the old type log, with this config we don't have any problems. Sure, on the first read we have duplicates, but then not anymore.

I know log is deprecated, this is the reason why I want to understand our problem.

4

Now I go back to log without any problems. Any ideas how I can solve the problem?