Hi,
I am using filebeat to send logs to logstash. Recently I encountered an issue that filebeat publishes the whole log file again to logstash which creates duplicate entries in elasticsearch/kibana.
Please see this filebeat log:
2017-03-20T12:06:12+05:30 INFO Non-zero metrics in the last 30s: filebeat.harvester.started=1 libbeat.logstash.call_count.PublishEvents=1 publish.events=3 filebeat.harvester.open_files=1 libbeat.logstash.published_and_acked_events=2 libbeat.publisher.published_events=2 registrar.states.update=3 registrar.writes=1 filebeat.harvester.running=1 libbeat.logstash.publish.write_bytes=638 libbeat.logstash.publish.read_bytes=35
2017-03-20T12:06:42+05:30 INFO No non-zero metrics in the last 30s
2017-03-20T12:07:12+05:30 INFO No non-zero metrics in the last 30s
2017-03-20T12:07:42+05:30 INFO No non-zero metrics in the last 30s
2017-03-20T12:07:43+05:30 INFO Harvester started for file: /var/log/radius/radius-rejected.log
2017-03-20T12:07:43+05:30 ERR Failed to publish events caused by: write tcp 127.0.0.1:33786->127.0.0.1:5044: write: connection reset by peer
2017-03-20T12:07:43+05:30 INFO Error publishing events (retrying): write tcp 127.0.0.1:33786->127.0.0.1:5044: write: connection reset by peer
2017-03-20T12:08:12+05:30 INFO Non-zero metrics in the last 30s: libbeat.logstash.published_but_not_acked_events=97 registrar.writes=1 libbeat.logstash.publish.read_bytes=1521 libbeat.logstash.publish.write_bytes=6001 libbeat.logstash.publish.write_errors=2 libbeat.logstash.published_and_acked_events=97 filebeat.harvester.running=1 publish.events=98 libbeat.publisher.published_events=97 libbeat.logstash.call_count.PublishEvents=2 registrar.states.update=98 filebeat.harvester.open_files=1 filebeat.harvester.started=1
2017-03-20T12:08:42+05:30 INFO No non-zero metrics in the last 30s
So it seems that there was some connection issue to logstash and then after sometime, the filebeat publishes the full log again ( see the publishes events ) to logstash which causes duplicate entries to display in elasticsearch / kibana.
Any idea how to resolve this issue?
Thank you.