# Logstash Elasticsearch plugin compare inputs

**URL:** <https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152>\
**Category:** Logstash\
**Created:** [June 7, 2021, 12:57pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152 "2021-06-07T12:57:49Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![Armen\_Petrosyan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armen_petrosyan/32/80330_2.png) [@Armen\_Petrosyan](https://discuss.elastic.co/u/Armen_Petrosyan)\
**Post date:** [June 7, 2021, 12:57pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/1 "2021-06-07T12:57:51Z")

</div>

I have logstash configured to have two inputs from different ES clusters

```
    input {
    elasticsearch {
        hosts => ["xxx:111"]
        ssl => true
        user => ""
        password => ""
        ca_file => ""
        index => "*"
        docinfo => true
    }
    elasticsearch {
        hosts => ["jjj:9200"]
        ssl => true
        user => ""
        password => ""
        ca_file => ""
        index => "*"
        docinfo => true
    }      
}
filter {
 mutate {
  remove_field => ["@version", "@timestamp"]
 }
}
output {
  stdout { codec => rubydebug }
}

```

I would like to compare \_source of each document between those inputs. In case if it is not the same - print it to the log.

Is there any way to do that? Maybe some specific compare plugin?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 7, 2021, 5:01pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/2 "2021-06-07T17:01:13Z")

</div>

> [@Armen\_Petrosyan](#):
>
> I would like to compare \_source of each document between those inputs. In case if it is not the same - print it to the log.

The events from each input are processed separately by logstash. You might be able to do it by using an elasticsearch input for one cluster, then using an elasticsearch filter to look up the same document in the other cluster.

---

<div class="post-metadata">

**Author:** ![Armen\_Petrosyan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armen_petrosyan/32/80330_2.png) [@Armen\_Petrosyan](https://discuss.elastic.co/u/Armen_Petrosyan)\
**Post date:** [June 8, 2021, 6:12am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/3 "2021-06-08T06:12:54Z")

</div>

Sound great, but can you please maybe share some config example for that, let's suppose \_id from the first cluster is the field I want to search in second cluster in order to find the doc for comparing it.

---

<div class="post-metadata">

**Author:** ![Armen\_Petrosyan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armen_petrosyan/32/80330_2.png) [@Armen\_Petrosyan](https://discuss.elastic.co/u/Armen_Petrosyan)\
**Post date:** [June 9, 2021, 3:20pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/4 "2021-06-09T15:20:04Z")

</div>

@Badger could you please provide some an example of how to do it?  
And does it mean that each document will be processed separately (checked/filtered)? If yes, it sounds like it might take a lot of time to check hundred of thousands doc and many pings to target cluster...

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 9, 2021, 3:50pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/5 "2021-06-09T15:50:18Z")

</div>

No, I cannot provide an example because I do not run elasticsearch. An API call to ES is going to be expensive compared to many other pipeline operations.

---

<div class="post-metadata">

**Author:** ![Armen\_Petrosyan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/armen_petrosyan/32/80330_2.png) [@Armen\_Petrosyan](https://discuss.elastic.co/u/Armen_Petrosyan)\
**Post date:** [June 10, 2021, 7:23am UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/6 "2021-06-10T07:23:02Z")

</div>

> No, I cannot provide an example because I do not run elasticsearch.

I understand, thanks

> [@Badger](#):
>
> then using an elasticsearch filter to look up the same document in the other cluster

Don't sure what does it mean? Could you maybe point me the the example how I can **look up** by id for example? And how to compare each field?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 10, 2021, 4:07pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/7 "2021-06-10T16:07:21Z")

</div>

There is an example of using an elasticsearch filter [here](https://discuss.elastic.co/t/logstash-elasticsearch-filter-lookup-no-results/272933).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2021, 4:07pm UTC](https://discuss.elastic.co/t/logstash-elasticsearch-plugin-compare-inputs/275152/8 "2021-07-08T16:07:39Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.