Reparsing same files

Elastic Stack Logstash
1

Hi Team,

I have one doubt:

Does reparsing of same log files having with same index name will save new records in elasticsearch every time we execute the logstash pipeline ?

If yes, then will these entry will reflect in saved search /visualization /dashboard of kibana?

Thanks

2

That solely depends on the id field of the ElasticSearch documents.

If you let ES generate it's own IDs or you generate random IDs yourself for each log, ES will treat them as different documents that happen to have the same content, and it will show on Kibana as extra records.

If you generate a preset ID for each log based on some existing log field (e.g. timestamp, if it exists) or other method, then ES should just overwrite the previous document and bump it's version number, resulting in no duplicate records.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.