# Transform with two input indices with different unique ids **URL:** \ **Category:** Elasticsearch\ **Tags:** painless, transforms\ **Created:** [October 8, 2024, 6:01pm UTC](https://discuss.elastic.co/t/transform-with-two-input-indices-with-different-unique-ids/368475 "2024-10-08T18:01:43Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Phil\_McLachlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_mclachlan/32/124424_2.png) [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\ **Post date:** [October 8, 2024, 6:01pm UTC](https://discuss.elastic.co/t/transform-with-two-input-indices-with-different-unique-ids/368475/1 "2024-10-08T18:01:43Z")
Hi, we have a transform with two input indices with different unique ids. One input index has a unique id of product\_pk, and another has product\_pk combined with catalog\_type. There are two possible catalog\_types: catalog and teyos. If I create a tranform pivoting on just product\_pk, then it seems to be missing documents from the index that have both types catalog and teyos. However, most of the documents just have a catalog type and most documents end up in the final index. I need all the documents to be in the final index that have any catalog type. Consequently, I tried pivoting on the product\_pk and the catalog\_type. This only gave me information from the second input index. I'm trying to combine data from both indices. Does anyone know what I have to pivot on and what can be done to get all documents with all info? We want a final index with product\_pks that are in both input indices only. One solution I can think of is to generate the first input index with both product\_pk and catalog\_type, thereby duplicating the data. This input index is already 1 Gig, and I don't want to double it's size. Also, this approach is not scalable, if we decide to add future catalog\_types. Any help would be appreciated. ---
**Author:** ![Phil\_McLachlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_mclachlan/32/124424_2.png) [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\ **Post date:** [October 11, 2024, 7:09pm UTC](https://discuss.elastic.co/t/transform-with-two-input-indices-with-different-unique-ids/368475/2 "2024-10-11T19:09:15Z")
I tried a proof of concept for another idea. Unfortunately, it didn't work either. I am still missing the same documents. The idea is this. I moved the catalog\_types into an array in the second input index, so it can be keyed in the product\_pk only. Then in the transform, I wrote a combine\_script that pulled out the catalog\_types from the document and put each element into a separate document copying the rest of the document with it. This resulted in the same output as before, with missing documents. I am now puzzled. ---
**Author:** ![Patrick\_Whelan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/patrick_whelan/32/135049_2.png) [@Patrick\_Whelan](https://discuss.elastic.co/u/Patrick_Whelan)\ **Post date:** [October 15, 2024, 2:13pm UTC](https://discuss.elastic.co/t/transform-with-two-input-indices-with-different-unique-ids/368475/3 "2024-10-15T14:13:21Z")
What do the data structures look like for the two different indices? It might be possible to attach a custom analyzer for the second index to split up the unique id from the category? [Create a custom analyzer | Elasticsearch Guide [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/analysis-custom-analyzer.html) Then both indices would have the same unique id field to pivot on. Another option would be a split processor if the fields are combined in some way: [Split processor | Elasticsearch Guide [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/split-processor.html) Alternatively, depending on the data structure, a query parameter attached in the Transform configuration may help: [Create transform API | Elasticsearch Guide [8.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-transform.html) ---
**Author:** ![Phil\_McLachlan](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phil_mclachlan/32/124424_2.png) [@Phil\_McLachlan](https://discuss.elastic.co/u/Phil_McLachlan)\ **Post date:** [October 16, 2024, 9:11pm UTC](https://discuss.elastic.co/t/transform-with-two-input-indices-with-different-unique-ids/368475/4 "2024-10-16T21:11:18Z")
Thanks Patrick for the suggestions. I don't think the solutions you provided would work for our situation. We want all the data that is in both indices with either catalog type, which is a separate field to the product pk. Although it is unimplemented yet, we have decided to make separate output indices for both catalog types. This means two transforms. One would be for each catalog type, and we would prepare separate input input indices for the second input index. In our first pass, we will do only one type of catalog type, and leave the other for another time.