# Problem with "joining" data **URL:** \ **Category:** Elasticsearch\ **Created:** [January 14, 2020, 11:19am UTC](https://discuss.elastic.co/t/problem-with-joining-data/214981 "2020-01-14T11:19:34Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Pieter\_Agenbag](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pieter_agenbag/32/4562_2.png) [@Pieter\_Agenbag](https://discuss.elastic.co/u/Pieter_Agenbag)\ **Post date:** [January 14, 2020, 11:19am UTC](https://discuss.elastic.co/t/problem-with-joining-data/214981/1 "2020-01-14T11:19:34Z")
I have a scenario where I have a bunch of indices (~100) each with thousands to several millions of documents each. My problem is each of these documents can be "linked" to one or more containers (Just a grouping name for simplicity). And I need to be able to filter documents in any of these indices based on a selected container. The obvious answer would be to carry the container name/id on each document , which works perfectly , except they can be linked or unlinked to containers frequently and the required update\_by\_query takes several hours ☹ I cant use parent child links , because parent child can only link in the same index (and can have only one parent and they have to be on the same shard) I tried indexing a separate document with and array of ids for each doc in the container ...and then using a terms lookup query, which also works great , but only for relatively small amounts of docs per container (I upped the max\_terms to 10mil to test 😮 ) I tried the same thing with several smaller arrays and doing a bool-should with all the subsets , but that takes even longer. Since I use sequential numbering for ids , I tried to save the containers grouping as a set of id ranges , but the groups end up being waaay too scattered for that approach (ending up with 300 000 range queries) A typical query I run against the indices returns in ~100ms ,but the fastest (with the terms lookup) I've been able to get with the container filter is 10-\>30s which makes the application unusably unresponsive (I need sub second) Does anyone have ANY ideas for me ? I'm on ES 7.5 . I can reindex and change mappings and go crazy with the data as long as I can get the performance. ---
**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\ **Post date:** [January 14, 2020, 5:01pm UTC](https://discuss.elastic.co/t/problem-with-joining-data/214981/2 "2020-01-14T17:01:28Z")
Hey, I am not a hundred percent sure, if this fits your use case, but with the release of 7.5, the [enrich processor](https://www.elastic.co/guide/en/elasticsearch/reference/7.5/ingest-enriching-data.html) has been released, that allows to enrich documents at index time with information from another index. Maybe this allows you to add some required fields to one of the indices? --Alex ---
**Author:** ![Pieter\_Agenbag](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pieter_agenbag/32/4562_2.png) [@Pieter\_Agenbag](https://discuss.elastic.co/u/Pieter_Agenbag)\ **Post date:** [January 15, 2020, 4:41am UTC](https://discuss.elastic.co/t/problem-with-joining-data/214981/3 "2020-01-15T04:41:07Z")
Thank for your reply . The enrich processor looks interesting and I'll definitely keep it in mind going forward , but unfortunately I dont think it'll help my current problem ... If I do go the route of saving the container/group name/id On the documents , then the problem is not with initial indexing , but continuously updating them afterwards to reflect any changes in linking. I have a workaround for now that would allow me to keep the number of documents per container down to a more manageable amount , allowing me to use the terms lookup method . ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 12, 2020, 4:41am UTC](https://discuss.elastic.co/t/problem-with-joining-data/214981/4 "2020-02-12T04:41:07Z")
Closed on February 12, 2020, 4:41am UTC This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.