# Enrich document with data from same index **URL:** \ **Category:** Elasticsearch\ **Created:** [December 20, 2023, 10:38am UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705 "2023-12-20T10:38:04Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![rickardo](https://avatars.discourse-cdn.com/v4/letter/r/9e8a1a/32.png) [@rickardo](https://discuss.elastic.co/u/rickardo)\ **Post date:** [December 20, 2023, 10:38am UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/1 "2023-12-20T10:38:04Z")
Hi, we got one index that we insert documents where one can follow e.g. a session and what is done, i.e.: document 1; sessionId = 17 type=created country=DE document 2: sessionId = 17 type=action now at insertion of doc 2 we want to enrich it with data from doc 1 (same sessionId), in this case add field country=DE to doc 2. A session is not long lived (secconds/minutes) and there will be a lot of newly created sessions. How can this be accomplished? I read a bit about Ingest pipelines but wondering if that is the way to go or if there are other better ways? ---
**Author:** ![RabBit\_BR](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rabbit_br/32/82261_2.png) [@RabBit\_BR](https://discuss.elastic.co/u/RabBit_BR)\ **Post date:** [December 20, 2023, 12:40pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/2 "2023-12-20T12:40:20Z")
Hi @rickardo I believe you've already read about the [Enrich pipeline](https://www.elastic.co/guide/en/elasticsearch/reference/current/ingest-enriching-data.html#enrich-policy) and I think that's one option. Another would be to have an isolated data enrichment service, in which your application requests enrichment before indexing the data. The second option is how we do it here at our company. ---
**Author:** ![rickardo](https://avatars.discourse-cdn.com/v4/letter/r/9e8a1a/32.png) [@rickardo](https://discuss.elastic.co/u/rickardo)\ **Post date:** [December 20, 2023, 2:05pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/3 "2023-12-20T14:05:06Z")
Thanks @RabBit_BR , I read that using Enrich pipeline will create an enrich index which I felt was a bit unclear of how it is maintained. In this usecase the "enrich" documents only need to be there for a short time and was concerned that it might grow quite large since we will have a lot of those coming in (many millions per month). Can one set time to live setting on the "enrich index"? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 20, 2023, 2:44pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/4 "2023-12-20T14:44:48Z")
> [@rickardo](#): > > I read that using Enrich pipeline will create an enrich index which I felt was a bit unclear of how it is maintained. If you need to update the data on the enrich index you need to use the execute API every time, enrich indices are best used for static data or data that is not update frequently. In your case it seems that you would need to frequently update it. The best way is to do that enrichment **before** indexing the data in Elasticsearch, but sometimes this is pretty hard to do or even not possible in a useful way. > [@rickardo](#): > > A session is not long lived (secconds/minutes) and there will be a lot of newly created sessions. Is the session id unique? Could the events like _created_ and _action_ happens too close for each other, like milliseconds? How are you indexing your data? ---
**Author:** ![rickardo](https://avatars.discourse-cdn.com/v4/letter/r/9e8a1a/32.png) [@rickardo](https://discuss.elastic.co/u/rickardo)\ **Post date:** [December 20, 2023, 3:07pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/5 "2023-12-20T15:07:52Z")
Thanks @leandrojmp ! ok, thought it was best suited for static data.. The sesssionId is unique over that session which normally would be seconds/minutes (1 - 10 documents). Events would not be miliisecons but more like seconds, it is manual user input. We havent done any index mapping for now if that is what you mean, just default. This is what the mapping looks like: "country": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 } } },sessionId": { "type": "text", "fields": { "keyword": { "type": "keyword", "ignore\_above": 256 } } } ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 20, 2023, 3:10pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/6 "2023-12-20T15:10:46Z")
Yeah, but how are you indexing the data? Are you using Logstash, Filebeat, a Custom script? Do you have any control over the source of your data? For example, how is it created? The enrich depends on how you are indexing your data, you would need to store the key value pair with the sessionId and country somewhere, and read it while indexing. ---
**Author:** ![rickardo](https://avatars.discourse-cdn.com/v4/letter/r/9e8a1a/32.png) [@rickardo](https://discuss.elastic.co/u/rickardo)\ **Post date:** [December 20, 2023, 3:20pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/7 "2023-12-20T15:20:40Z")
Yes, we have control, we are saving through org.springframework.data.elasticsearch.repository.ElasticsearchRepository. The problem for us is that we can scale our services and thus we would have to first do a lookup to ES before enriching. If we had just one entry point i guess it would be easy to cache the first entry. Unless we had a shared cache/store mechanism outside ES but the comlicates things. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [December 20, 2023, 3:35pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/8 "2023-12-20T15:35:30Z")
> [@rickardo](#): > > Unless we had a shared cache/store mechanism outside ES but the comlicates things. Yeah, but this is the right approach in this case. You would need to store it somewhere else, maybe a cache database like Redis/Memcached and query it every time you receive a new document, before indexing in Elasticsearch. The enrich processor will not work in this case. ---
**Author:** ![rickardo](https://avatars.discourse-cdn.com/v4/letter/r/9e8a1a/32.png) [@rickardo](https://discuss.elastic.co/u/rickardo)\ **Post date:** [December 20, 2023, 3:37pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/9 "2023-12-20T15:37:31Z")
Ok, good to know! Thanks for your quick response! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 17, 2024, 3:37pm UTC](https://discuss.elastic.co/t/enrich-document-with-data-from-same-index/349705/10 "2024-01-17T15:37:32Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.