Hi!
-
We have many servers with a Tomcat in each server, with several applications in each server, and we want to save the applications log files in Elastic, so in each server we also have a Filebeat.
-Between Filebeat and Elasticsearch, we have a two nodes Logstash cluster -
The log files each application writes rotate every 10 MB (It is log4j which makes them rotate, they rotate by renaming the files, inode remains always the same), so we have up to 11 files, for example "trace.log", "trace.log.1", "trace.log.2"... "trace.log.10"
-
filebeat.yml config file "type" is:
- type: log enabled: true paths: - /logs/trace/trace.log* multiline.pattern: '^\[[0-9]{2}\/[0-9]{2}\/[0-9]{4}' multiline.negate: true multiline.match: after fields: tipo: standard infra: ob -
These servers (machines) are never switched off, Tomcat never stops, Filebeat never stops
-
Then we have 3 Elasticsearch nodes and 2 Logstash + Kibana nodes
- Elasticsearch, logstash and kibana nodes are switched off at night, and stay turned off for all night (like 12 hours) -
While Elasticsearch, Logstash and Kibana are working (switched on) everything works like a charm, even when log files rotate.
-
However, and this is the problem, when a log file rotates while Elasticsearch, Logstash and Kibana are down (at nigh), all the rotated files are ingested again (so everything gets duplicated)
-
For example, we have already in Elasticsearch all the data for "trace.log", "trace.log.1", "trace.log.2"... "trace.log.10" and trace.log at night rotates, when Elasticsearch, Kibana and Logstash start, all those files "trace.log", "trace.log.1", "trace.log.2"... "trace.log.10" get ingested AGAIN in Elasticsearch, duplicanting all those documents.
Why is this happening? Amost every night at least one file gets rotated. On weekend many files do!
- It is not a problem of Filebeat sending "at least once delivery" it is duplicating 11 complete files!!
Thanks in advance!!