# Efficient and fault-tolerant way to "merge" two indexes

**URL:** <https://discuss.elastic.co/t/efficient-and-fault-tolerant-way-to-merge-two-indexes/234069>\
**Category:** Elasticsearch\
**Created:** [May 24, 2020, 9:33pm UTC](https://discuss.elastic.co/t/efficient-and-fault-tolerant-way-to-merge-two-indexes/234069 "2020-05-24T21:33:24Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![gabriel.goncalves](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gabriel.goncalves/32/47574_2.png) [@gabriel.goncalves](https://discuss.elastic.co/u/gabriel.goncalves)\
**Post date:** [May 24, 2020, 9:33pm UTC](https://discuss.elastic.co/t/efficient-and-fault-tolerant-way-to-merge-two-indexes/234069/1 "2020-05-24T21:33:24Z")

</div>

Hi there. I'd like and efficient and fault tolerant (connection problems, failed operations etc) way to do the following:

I need to "merge" two indexes. Lets call them A and B being merged into C;

For every entry in A, there are 1 or more entries in B which are associated with it. We could think of A as a flights and B its passengers, for instance. Let 's describe an example of such a group by (A', [B'1, B'2,...B'n]);

Each entry in C will consist of:  
-Full data of an entry in A  
-Full data of an entry in B  
-A few more computed fields - lets call them Cf ;  
Each entry in A will appear multiple times in C, one for each entry in B associated with it;  
For the group (A', [B'1, B'2,...B'n]), the entries in C would be:  
(A',B'1,Cf1), (A',B'2,Cf2),..,(A',B'n,Cfn)

Things I'm considering: Best way to scroll and search through A and B to produce the entries for C. If in this case would be use useful to create auxiliary fields en A and/or B to register which items were already processed, which ones  
are being processed and etc and speed up the scrolling. If its worth to use sort \_doc in the scrolling for the speed or the fact that I won't be able to use  
"search\_after" with '\_doc' makes it not worth. The way to organize the process such that the work can be divided in threads and that it automatically deals  
with failed create/index operations made withing bulks etc

The indexes A and B have around 20 million and 30 million entries respectivelly. Each entry has approx. 10kb in size.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 21, 2020, 9:33pm UTC](https://discuss.elastic.co/t/efficient-and-fault-tolerant-way-to-merge-two-indexes/234069/2 "2020-06-21T21:33:32Z")

</div>

Closed on June 21, 2020, 9:33pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.