# Elasticsearch filter in logstash = random results

**URL:** <https://discuss.elastic.co/t/elasticsearch-filter-in-logstash-random-results/288718>\
**Category:** Logstash\
**Created:** [November 9, 2021, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-filter-in-logstash-random-results/288718 "2021-11-09T07:52:23Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rnx](https://avatars.discourse-cdn.com/v4/letter/r/6de8d8/32.png) [@Rnx](https://discuss.elastic.co/u/Rnx)\
**Post date:** [November 9, 2021, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-filter-in-logstash-random-results/288718/1 "2021-11-09T07:52:23Z")

</div>

Hi,  
I'm using Elasticsearch filter plugin in logstash to enrich records and calculate time difference between two events from different logs. It works, but randomly - it returns search results for some records, but not for all of them. The pain is, it's without any pattern what is wrong.  
Logstash conf:

```auto
                elasticsearch {
                        hosts => ["https://10.20.30.40:9200"]
                        index => "e2e-strsuni-*"
                        query_template => "/etc/logstash/conf.d/query.json"
                        fields => {
                        # "saptype" => "saptype"
                                "file_id" => "file_id"
                                "E2E_SFTP" => "E2E_SFTP"
                                "E2E_DP" => "E2E_DP"
                                "date_meta" => "date_meta"
                                "date_sftp" => "date_sftp"
                                "@timestamp" => "date_dp"
                        }
                        user => "some_funky_password"
                        password => "some_user"
                        ca_file => "/etc/logstash/conf.d/elasticsearch-ca.pem"
                        #ssl => true
                }

```

query.json:

```auto
{"size":1,
"sort":[{"@timestamp":"desc"}],
"query":{"simple_query_string":{"query":"ext_id:%{[ext_id]}"}}}

```

So it should look like this 3 records - columns **E2E\_SFTP, E2E\_DP, E2E\_RESPONSE** and **date\_processed** should be filled as they inherit "true" and time filed from previous one.

 ![Screenshot 2021-11-09 at 8.43.46](https://us1.discourse-cdn.com/elastic/original/3X/e/4/e4605bc086e28b8114a25ace3e7b17a4f09cc259.png)

If it doesn't work, it looks like this:

 ![Screenshot 2021-11-09 at 8.42.28](https://us1.discourse-cdn.com/elastic/original/3X/1/d/1d3656708667aef9ebd2824e4555123b2c29a039.png)

There is no pattern in time, or id or anything, simply some of records are "found" via filter plugin and then are processed as they should be, some of them are not so they're indexed without "inherited" fields, and there is no record in any log why is this behavior occurring. Any ideas what to check or search for?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 7, 2021, 7:52am UTC](https://discuss.elastic.co/t/elasticsearch-filter-in-logstash-random-results/288718/2 "2021-12-07T07:52:59Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.