Import addresses into Elasticsearch using Logstash

Elastic Stack Logstash
1

Hi,

I'd like to import addresses into ES using Logstash. The thing is, I'd like to filter out the addresses that already exist in the ES index but I can't find a ES filter plugin that allows me to first check if the address already exists in ES.

Any ideas? Or is Logstash the wrong approach?

2

You could use the fingerprint filter plugin to generate a fingerprint that you then use as the document id. Used together with the appropriate action in the elasticsearch output, you can then e.g. upsert (insert or update) the existing document, or just keep the first copy indexed.

3

Thanks for your answer.

The problem is (and I forgot that to write, sorry) that the ES addresses are company data and the imported ones are customer data, therefore consistent hashes don't work. That's why I first thought a preceding query in a filter would solve my problem.

4

I haven't tested this idea yet, but you could e.g. use the elasticsearch filter plugin with a query template that matches your stored addresses, and then drop the new event if a match was found.

5

Thanks for that input.

I understand what you mean but I'm a newbie in the ELK stack and therefore have no clue how to implement this approach.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.