# ES query to check the existence of a document\_id? **URL:** \ **Category:** Logstash\ **Created:** [May 29, 2020, 4:39am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854 "2020-05-29T04:39:53Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Nani\_20](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Post date:** [May 29, 2020, 4:39am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/1 "2020-05-29T04:39:53Z")
I'm using a **file input** in logstash to input files but I encounter duplicates time to time. Currently, I'm using the following the below options in **ES output plugin** to avoid duplicates. > document\_id =\> "%{message}" > action =\> "create" But, I see these duplicates passing through the **filter plugin** which has lot of ruby operations. Is there any way to do this in an ES filter plugin to query efficiently and avoid these duplicates passing through my ruby operations. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2020, 4:45am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/2 "2020-05-29T04:45:04Z")
What does the rest of the elasticsearch output configuration look like? Can you show us and example of two duplicate events that have been inserted into Elasticsearch, including the \_id and \_index? ---
**Author:** ![Nani\_20](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Post date:** [May 29, 2020, 4:59am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/3 "2020-05-29T04:59:29Z")
@Christian_Dahlqvist Thanks for your reply. Find below my **ES output plugin**. **ES output plugin** ``` elasticsearch{ hosts => ["http://localhost:9200"] index => "sample_index" #document_id => "%{message}" #action => "create" } ``` I'm using logstash **file input** to index various log files available in the input location. These logfiles are generated by log4j in various systems and are fetched using a cron job to the input location. By default, log4j creates the backup of a log file after a size limit, so logstash receives duplicate logs from time to time. These log lines have generally a timestamp and other information about the events. I'm using full message as an ID to avoid duplicates. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2020, 5:10am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/4 "2020-05-29T05:10:38Z")
This does not correspond to what you posted earlier. Using a full message as an id seems like a bad idea as it could contain illegal or unsuitable characters. It is probably better to create and use a hash. I would recommend reading [this blog post](https://www.elastic.co/blog/efficient-duplicate-prevention-for-event-based-data-in-elasticsearch) as well as [this one](https://www.elastic.co/blog/logstash-lessons-handling-duplicates). ---
**Author:** ![Nani\_20](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Post date:** [May 29, 2020, 8:24am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/5 "2020-05-29T08:24:28Z")
Sorry my bad, I have corrected the config file now. I think I had different file copied initially. Anyways, I would like to use the **entire message as an ID**. As you suggested, I think it would be ideal to generate **UUID** using **fingerprint filter** and then use it to avoid duplicates. I am I right here? But, this will still allow duplicates to enter the other filter plugins right. Is there any way I can avoid/drop these duplicates right after creating the ID by **fingerprint filter** and check its existence in ES instead of waiting till **ES output plugin**. This will save lot of time and effort. Please refer to this [post](https://discuss.elastic.co/t/how-to-avoid-duplicates-before-entering-the-filter-plugin/234735/3) for your reference ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2020, 8:49am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/6 "2020-05-29T08:49:45Z")
If you associate a UUID with the event you need to do this directly at the source, e.g. in filebeat, for it to work. There is as far as I know no way to eliminate duplicates within the pipeline before the data gets to Elasticsearch. ---
**Author:** ![Nani\_20](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Post date:** [May 29, 2020, 8:50am UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/7 "2020-05-29T08:50:56Z")
I'm afraid that won't be possible as I'm reading the files using **file input**. So, I have to do this in the ES output plugin itself right? ---
**Author:** ![Nani\_20](https://avatars.discourse-cdn.com/v4/letter/n/4491bb/32.png) [@Nani\_20](https://discuss.elastic.co/u/Nani_20)\ **Post date:** [May 29, 2020, 12:36pm UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/8 "2020-05-29T12:36:00Z")
@Christian_Dahlqvist Could you please let me know of any alternative for **file input**? Thanks in advance ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [May 29, 2020, 1:19pm UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/9 "2020-05-29T13:19:59Z")
> [@Christian\_Dahlqvist](#): > > There is as far as I know no way to eliminate duplicates within the pipeline before the data gets to Elasticsearch. What I was suggesting was to generate the document\_id, using fingerprint or something, then use an elasticsearch filter to check whether the document exists. If it does, drop the event, otherwise, continue enriching the event and send it to the elasticsearch output. The question was (or should have been) how to use an elasticsearch filter to check for the existence of a given document\_id in an index. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [May 29, 2020, 1:29pm UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/10 "2020-05-29T13:29:08Z")
Given that approach will result in a query per document it will be very slow and not scale well. There is s also the issue around indexed events not being searchable immediately but with a low throughput that may be less of an issue. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 26, 2020, 1:29pm UTC](https://discuss.elastic.co/t/es-query-to-check-the-existence-of-a-document-id/234854/11 "2020-06-26T13:29:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.