# Retrieve documents matching criteria along with missing documentIds **URL:** \ **Category:** Elasticsearch\ **Tags:** kql-kibana-query-language, eql-elastic-query-language\ **Created:** [September 19, 2022, 7:31am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665 "2022-09-19T07:31:13Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![priyakc](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@priyakc](https://discuss.elastic.co/u/priyakc)\ **Post date:** [September 19, 2022, 7:31am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/1 "2022-09-19T07:31:13Z")
I have list of document ids. When I query Elasticsearch, In result I need document ids which are matching my criteria and documentIds for which documents do not exist in Elasticsearch. **Background on use case** I need to search for missing reports for sites. DocumentId of report is the siteId. A report is missing if its reportState is NEW (not submitted) or if it doesn't exist in Elasticsearch. **Expected Request and Response** Request: all sites i.e documentIds in request to Elasticsearch [ABC1, ABC2, ABC3] Documents in Elasticsearch [ABC1 with reportState NEW, ABC2 with reportState SUBMITTED, ABC3 document doesn't exist with this documentId] Now I want to build query X such that my response will retrieve below documentIds. [ABC1, ABC3] ```auto { "query": { "bool": { "should": [ { "term": { "reportState": "NEW" } }, { //Build something like below specifying document doesn't exist "must_not": { "exists": { "field": "reportState" } } } ] } } } ``` **Available Alternative** I found an approach using mget on [Check documents not existing at elasticsearch - Stack Overflow](https://stackoverflow.com/questions/59213885/check-documents-not-existing-at-elasticsearch) but I require to paginate over results, hence looking for a query. ---
**Author:** ![anime\_lover](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/anime_lover/32/100265_2.png) [@anime\_lover](https://discuss.elastic.co/u/anime_lover)\ **Post date:** [September 19, 2022, 11:04am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/2 "2022-09-19T11:04:12Z")
> [@priyakc](#): > > Elasticsearch can you please write question with example...where we can also operate on kibana to find solution ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 19, 2022, 11:53am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/3 "2022-09-19T11:53:43Z")
A query can never return documents that do not exist, so I do not see how you can build a query to return what you describe. If you knew you expected these two documents you could write a query to return the first one and deduce the other one is not present in the client application. ---
**Author:** ![priyakc](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@priyakc](https://discuss.elastic.co/u/priyakc)\ **Post date:** [September 19, 2022, 12:08pm UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/4 "2022-09-19T12:08:23Z")
I updated the question with sample query. ---
**Author:** ![priyakc](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@priyakc](https://discuss.elastic.co/u/priyakc)\ **Post date:** [September 19, 2022, 12:13pm UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/5 "2022-09-19T12:13:11Z")
I cannot do that. With your logic Request: [ABC1, ABC2, ABC3] Elasticsearch Response: [ABC1] By deducing other ones are not present [ABC2, ABC3], computed response becomes [ABC1, ABC2, ABC3] Whereas expected response is [ABC1, ABC3]. The issue here is if I fire a query with just one criteria ReportState=NEW, I do not know whether ABC2 didn't match my query or whether it didn't even exist. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 19, 2022, 12:23pm UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/6 "2022-09-19T12:23:28Z")
A query can not return a document that does not exist, so you need to find a way to handle this in your application logic or perhaps run multiple queries. ---
**Author:** ![priyakc](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@priyakc](https://discuss.elastic.co/u/priyakc)\ **Post date:** [September 19, 2022, 12:31pm UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/7 "2022-09-19T12:31:34Z")
Ofcourse I can handle this in application logic but its going to be a heavy operation. I am looking for an optimised approach to achieve this. Using mget is the only way I could find so far, but I need pagination as well over missing reports. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 19, 2022, 1:40pm UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/8 "2022-09-19T13:40:28Z")
Can you perhaps initially create documents matching all reports you expect and let these have a special field that indicate that they do not yet exist. When you submit a report you overwrite this with the proper document. This way you might be able to adjust your search criteria to return these together with the matching ones. ---
**Author:** ![priyakc](https://avatars.discourse-cdn.com/v4/letter/p/b3f665/32.png) [@priyakc](https://discuss.elastic.co/u/priyakc)\ **Post date:** [September 22, 2022, 10:33am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/9 "2022-09-22T10:33:36Z")
We have DynamoDB as primary storage, we would require to build a complex mechanism to create documents by default for our use case. That would be last option for us. Looking to solve using Elasticsearch itself. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 22, 2022, 10:51am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/10 "2022-09-22T10:51:39Z")
That is as far as I know not possible. ---
**Author:** ![Mahi13](https://avatars.discourse-cdn.com/v4/letter/m/7ea924/32.png) [@Mahi13](https://discuss.elastic.co/u/Mahi13)\ **Post date:** [September 25, 2022, 3:01am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/11 "2022-09-25T03:01:47Z")
Hi, Try wrapping must\_not query with another bool query because must\_not query works only with bool query ```auto { "query": { "bool": { "should": [ { "term": { "reportState": "NEW" } }, { //Build something like below specifying document doesn't exist "bool":[ "must_not": { "exists": { "field": "reportState" } } ] } ] } } } ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 23, 2022, 3:02am UTC](https://discuss.elastic.co/t/retrieve-documents-matching-criteria-along-with-missing-documentids/314665/12 "2022-10-23T03:02:16Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.