Logstash Elasticsearch plugin compare inputs

Elastic Stack Logstash
1

I have logstash configured to have two inputs from different ES clusters

    input {
    elasticsearch {
        hosts => ["xxx:111"]
        ssl => true
        user => ""
        password => ""
        ca_file => ""
        index => "*"
        docinfo => true
    }
    elasticsearch {
        hosts => ["jjj:9200"]
        ssl => true
        user => ""
        password => ""
        ca_file => ""
        index => "*"
        docinfo => true
    }      
}
filter {
 mutate {
  remove_field => ["@version", "@timestamp"]
 }
}
output {
  stdout { codec => rubydebug }
}

I would like to compare _source of each document between those inputs. In case if it is not the same - print it to the log.

Is there any way to do that? Maybe some specific compare plugin?

2

The events from each input are processed separately by logstash. You might be able to do it by using an elasticsearch input for one cluster, then using an elasticsearch filter to look up the same document in the other cluster.

3

Sound great, but can you please maybe share some config example for that, let's suppose _id from the first cluster is the field I want to search in second cluster in order to find the doc for comparing it.

4

@Badger could you please provide some an example of how to do it?
And does it mean that each document will be processed separately (checked/filtered)? If yes, it sounds like it might take a lot of time to check hundred of thousands doc and many pings to target cluster...

5

No, I cannot provide an example because I do not run elasticsearch. An API call to ES is going to be expensive compared to many other pipeline operations.

6

No, I cannot provide an example because I do not run elasticsearch.

I understand, thanks

Don't sure what does it mean? Could you maybe point me the the example how I can look up by id for example? And how to compare each field?

7

There is an example of using an elasticsearch filter here.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.