# Continuously merging documents from multiple indexes
**URL:** https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953
**Category:** Elasticsearch
**Created:** [July 29, 2021, 8:16am UTC](https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953 "2021-07-29T08:16:43Z")
**Posts on this page:** 4
**Page:** 1
### Author:  [@roll](https://discuss.elastic.co/u/roll)
#### Post date: [July 29, 2021, 8:16am UTC](https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953/1 "2021-07-29T08:16:43Z")
Hi all,
I'm fairly new to elasticsearch and have a question regarding merging of documents and removing duplicates.
I got the following data structure:
```auto
topic1:
{
"_id": “a12345",
"userId" : "x123”,
“externalId”: “0987654",
"firstName" : "first name",
"lastName" : "last name",
"birthDate" : -157770000000,
"vipFlag" : "1",
"email" : "user@domain.org",
"mobile" : "+12345678",
"zipCode" : "12345",
"city" : "City name"
}
topic2:
{
"_id": “b12345",
"userId" : "x123”,
“externalId”: “0987654",
"firstName" : "first name",
"lastName" : "last name",
"birthDate" : -157770000000,
"email" : "user@domain.org",
"mobile" : "+12345678",
"zipCode" : "12345",
"city" : "City name"
}
topic3:
{
"_id": “c12345",
“externalId”: “0987654",
"firstName" : "first name",
"lastName" : "last name",
"email" : "user@domain.org",
"mobile" : "+12345678",
"zipCode" : "12345",
}
topic4:
{
"_id": “b12345",
"userId" : "x123”,
"firstName" : "first name",
"lastName" : "last name",
"birthDate" : -157770000000,
"email" : "user@domain.org",
"mobile" : "+12345678",
"zipCode" : "12345",
"city" : "City name"
}
```
The information is pushed continuously into ES from the topics in kafka.
I’m considering using Kafka Elasticsearch Sink Connector to store the data in separate indexes per topic and merge the values based on the merging rules into a new index that will be used for querying.
Following values should be used for merging the documents from topics:
- topic1 with topic2: userId
- topic1 with topic4: userId
- topic1\_2\_4 with topic3: externalId
As a result I’m expecting one topic with either parent-child relation or with merged fields (whatever is easier)
Is it doable in elasticsearch or should I write a an external kafka consumer + elasticsearch indexer application to merge and insert the data?
If it's doable in elasticsearch, what would be the right way to solve this?
---
### Author:  [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)
#### Post date: [July 29, 2021, 9:25am UTC](https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953/2 "2021-07-29T09:25:07Z")
You can use a continuous [transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/transforms.html) for this, however you need 2, one that groups by `userId` and a 2nd one that merges the output of the 1st transform with topic3 by grouping on `externalId`.
However for a continuous transform all your source indexes require timestamp field, if you don't have one you might [add one using an ingest timestamp](https://discuss.elastic.co/t/dec-12th-2018-en-elasticsearch-automatically-adding-a-timestamp-to-documents/159314). You also need this to connect transform 1 with transform 2.
For picking the right values for the output, e.g. `email` you can use a script via `scripted_metric` (spoiler: in a future release this will become easier). Have a look at the provided docs, in the sub sections you will find some painless examples, e.g. the one for [top\_hits](https://www.elastic.co/guide/en/elasticsearch/reference/current/transform-painless-examples.html#painless-top-hits).
---
### Author:  [@roll](https://discuss.elastic.co/u/roll)
#### Post date: [July 30, 2021, 10:50am UTC](https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953/3 "2021-07-30T10:50:38Z")
Thanks for your help @Hendrik_Muhs!
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 27, 2021, 10:50am UTC](https://discuss.elastic.co/t/continuously-merging-documents-from-multiple-indexes/279953/4 "2021-08-27T10:50:57Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.