Will logstash duplicate already indexed data in elasticsearch?

(Brook Hutchinson) #1

Windows 7
elasticsearch v1.6.0
logstash v1.5.2
kibana v4.1.0

Right now I am just working on a proof of concept locally and am not using a Logstash Forwarder. I am manually running logstash from the DOS Prompt to load data from an Apache Access Log into elasticsearch

Assume that I already have data indexed in elasticsearch. I do not want duplicate documents in elasticsearch. If I use logstash to an Apache Log File, if logstash/elasticsearch determines that the data from the log file already exists in elasticsearch, will the already indexed data in elasticsearch get duplicated or will the already indexed data be deleted and then re-added to elasticsearch (no duplication)?

(Magnus B├Ąck) #2

Logstash doesn't care what's already in Elasticsearch. The file input does however track how much of a file that has been processed which normally prevents duplication of input data.

(system) #3