# Enrich policy with integrated sort/search query **URL:** \ **Category:** Elasticsearch\ **Created:** [March 27, 2020, 7:53am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349 "2020-03-27T07:53:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![winlamp](https://avatars.discourse-cdn.com/v4/letter/w/cdc98d/32.png) [@winlamp](https://discuss.elastic.co/u/winlamp)\ **Post date:** [March 27, 2020, 7:53am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/1 "2020-03-27T07:53:53Z")
Hi, I'm trying to integrate a search query into my enrich policy. This step is required since I only require the most recent data from my index. Therefore i would like to do a sort based on the index @timestamp. I already managed to do a search that returns the field as required. It returns the latest entries for "Device-1" based on @timestamp. ```auto GET my-enrich-index-*/_search { "query": { "match": { "device.internal_name": "Device-1" } }, "sort": [ { "@timestamp": { "order": "desc" } } ], "size": 1 } ``` Now I need to integrate it into my enrich policy: ```auto PUT /_enrich/policy/MyDevice-policy { "match": { "indices": "my-enrich-index-*", "match_field": "device.internal_name", "enrich_fields": "serial_number" } } ``` Right now the enrich policy returns the first ever value ingested. No matter what I try to do it will not return the latest value from my-enrich-index-\*. Please advise of how to integrate my search into my enrich policy. Thx a lot! ---
**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\ **Post date:** [March 28, 2020, 4:16am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/2 "2020-03-28T04:16:19Z")
I don't think this is possible / how it is supposed to work. Taking the example from the docs for [exact match](https://www.elastic.co/guide/en/elasticsearch/reference/current/match-enrich-policy-type.html) this is using a `term` query. Since you need to set up an [enrich index](https://www.elastic.co/guide/en/elasticsearch/reference/current/enrich-setup.html) explicitly anyway I would create that without duplicates. If you use the unique matching field as the `_id` of of the document, you'll only have the current ones in there and don't have to worry about sorting any more. Also for performance reasons I'd keep this index as minimal as possible and keep historic values in another index (if needed). ---
**Author:** ![winlamp](https://avatars.discourse-cdn.com/v4/letter/w/cdc98d/32.png) [@winlamp](https://discuss.elastic.co/u/winlamp)\ **Post date:** [March 28, 2020, 8:08am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/3 "2020-03-28T08:08:30Z")
Ok. that's bad. My question right now is the following: If I use my unique ID as \_id I can't use it anymore since I have to use the same field name (enrich index and to be enriched index) to reference it, right?. In my to be enriched index the \_id field is something completely different since it is used for another use case. So how can I reference from my to be enriched index to the enrich index when the fields are named differently? thx again! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 28, 2020, 8:56am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/4 "2020-03-28T08:56:54Z")
> [@winlamp](#): > > Ok. that's bad. My question right now is the following: If I use my unique ID as \_id I can't use it anymore since I have to use the same field name (enrich index and to be enriched index) to reference it, right?. I do not understand. You keep the structure of the document as it is, but set the document ID to the the unique identifier for the device, e.g. `device.internal_name`. Every time a new document related to a specific device with that id comes in it will overwrite any existing version. You therefore keep only the most recent version for each device, which means that your query will always return just one document and you do not need the sort and size clauses. If you want to keep track of all the state changes, you can write all changes to a different index where you let Elasticsearch set the document id. ---
**Author:** ![winlamp](https://avatars.discourse-cdn.com/v4/letter/w/cdc98d/32.png) [@winlamp](https://discuss.elastic.co/u/winlamp)\ **Post date:** [March 28, 2020, 9:27am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/5 "2020-03-28T09:27:09Z")
Ok Christian! Since I'm a bit of a newbie please give me a hint of how do I assign the "\_id" when using an ingestion pipeline? I searched the ES reference and I haven't found the assistance to do that on my own. Thx! ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [March 28, 2020, 9:49am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/6 "2020-03-28T09:49:33Z")
You should be able to change [this example](https://www.elastic.co/guide/en/elasticsearch/reference/7.6/accessing-data-in-pipelines.html#accessing-metadata-fields) to set the field based on one of the fields in the document. ---
**Author:** ![winlamp](https://avatars.discourse-cdn.com/v4/letter/w/cdc98d/32.png) [@winlamp](https://discuss.elastic.co/u/winlamp)\ **Post date:** [March 28, 2020, 9:50am UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/7 "2020-03-28T09:50:21Z")
Thx! ---
**Author:** ![winlamp](https://avatars.discourse-cdn.com/v4/letter/w/cdc98d/32.png) [@winlamp](https://discuss.elastic.co/u/winlamp)\ **Post date:** [March 28, 2020, 2:04pm UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/8 "2020-03-28T14:04:48Z")
I managed to use the \_id field as storage for my unique intensifier. Data is pared correctly via my ingest pipeline. The only issue I encounter now: if I update the file and filebeat ingests it again, I see no change in my index. Even the timestamp doesn't change from the initial ingestion. On the other hand, if i change the field from \_id to something else it works as advertised. multiple version separated by the ingestion timestamp. Any idea what I do wrong? Thx again for your time! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [April 25, 2020, 2:05pm UTC](https://discuss.elastic.co/t/enrich-policy-with-integrated-sort-search-query/225349/9 "2020-04-25T14:05:00Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.