Filebeat sending duplicates events

Elastic Stack Beats
1

Hi,

Im running Elastic/Logstash/Filebeat v 7.15. I have beats installed on a few debian 9 machines, shipping log files. Everything is working well, except that we get duplicate entries. Seems like filebeat keep reading the log file even after it reached EOF. Sometimes its triplicating. I tried every possible ignore/close setting, but the issue persists.

The log files in question never updates. They are written once (2000 lines per log file), and remain unchanged. I should point out though that we have a lot log files in that directory. Its about 100k files. We receive on an average 60 new log files per minute during work hours.

Below are the relevent lines which i currently have in my beats config

- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    #- /var/log/*.log
    - /var/Vendorlogs/*
    #- /var/tmp/*
    #- c:\programdata\elasticsearch\logs\*
  ignore_older: 5m
  close_eof: true
  clean_inactive: 1h
  file_identity.inode_marker.path: /var/venderlog/.filebeat-marker
  #close_timeout: 2m
2

Hi, just wanted to bump this, and also add some addtional info which perhaps might help. Below is the output from log.json for a particular log file (weblog-1637856000-247698-0) which duplicated. I could see that the inode marker does not change.

{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":0,"updated":[281470681743360,18446744011573954816],"cursor":null,"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280187196233751,1637858551],"cursor":null,"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280187250213681,1637858551],"cursor":{"offset":74740},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280187313740456,1637858551],"cursor":{"offset":183405},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"cursor":{"offset":227657},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"},"ttl":1800000000000,"updated":[280187452949851,1637858551]}}
{"k":"filestream::.global::inode_marker::980556-","v":{"cursor":{"offset":265674},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"},"ttl":1800000000000,"updated":[280186580131151,1637858552]}}
{"k":"filestream::.global::inode_marker::980556-","v":{"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"},"ttl":1800000000000,"updated":[280186685760804,1637858552],"cursor":{"offset":284743}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280186790648341,1637858552],"cursor":{"offset":319862},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280186931880861,1637858552],"cursor":{"offset":348329},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280187003885190,1637858552],"cursor":{"offset":385380},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280187147221205,1637858552],"cursor":{"offset":445486},"meta":{"identifier_name":"inode_marker","source":"/var/venderlog/weblog-1637856000-247698-0"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"cursor":{"offset":482568},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"},"ttl":1800000000000,"updated":[280187280896918,1637858552]}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280187382709489,1637858552],"cursor":{"offset":540937},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280186532479640,1637858553],"cursor":{"offset":599132},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"},"ttl":1800000000000,"updated":[280186659160033,1637858553],"cursor":{"offset":629849}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280186754788315,1637858553],"cursor":{"offset":671509},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280186900275306,1637858553],"cursor":{"offset":723603},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"ttl":1800000000000,"updated":[280186962684109,1637858553],"cursor":{"offset":790959},"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"}}}
{"k":"filestream::.global::inode_marker::980556-","v":{"meta":{"source":"/var/venderlog/weblog-1637856000-247698-0","identifier_name":"inode_marker"},"ttl":1800000000000,"updated":[280187173897433,1637858553],"cursor":{"offset":839539}}}
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.