# Comparing two fields between two different index and output result into a third index

**URL:** <https://discuss.elastic.co/t/comparing-two-fields-between-two-different-index-and-output-result-into-a-third-index/286535>\
**Category:** Logstash\
**Created:** [October 12, 2021, 6:23pm UTC](https://discuss.elastic.co/t/comparing-two-fields-between-two-different-index-and-output-result-into-a-third-index/286535 "2021-10-12T18:23:22Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![test\_tester](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/test_tester/32/79990_2.png) [@test\_tester](https://discuss.elastic.co/u/test_tester)\
**Post date:** [October 12, 2021, 6:23pm UTC](https://discuss.elastic.co/t/comparing-two-fields-between-two-different-index-and-output-result-into-a-third-index/286535/1 "2021-10-12T18:23:22Z")

</div>

Hello!

I got a requirement whereby I am required to match two fields from two different files and output the result into a csv file.

Prior to this train of thought, i did a simple python script to do the matching logic between 2 excel files and output the result into a csv file, but well, it took roughly 10 hours and its not optimal speed wise.

Moving along, I then thought of trying it via logstash/Elasticsearch and have since been able to create my .conf pipeline files and ingest the data into two indexes.

So my problem starts here, I need to match/compare two fields (the field values are an exact match) from two different indexes and output the result to another index, so like 3 index.

E.g.

Index A - contains a field called indexA\_txn\_id  
Index B - contains a field called indexB\_txn\_id

Both fields have the same values.

Once a match is found, take indexA\_txn\_id and flush it to a new index, Index C.

Any ideas how I can go about this? Have considered stuff like aggregate or even getting rid of the index C idea completely and replacing it with a new field, something like, if indexA\_txn\_id == indexB\_txn\_id, add\_field ismatch\_txn\_id = 'matched'.

Any help or advise would be greatly appreciated. If it is impossible to do this at logstash layer, do recommend me an alternative if you have one.

---

<div class="post-metadata">

**Author:** ![AquaX](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aquax/32/92006_2.png) [@AquaX](https://discuss.elastic.co/u/AquaX)\
**Post date:** [October 12, 2021, 7:10pm UTC](https://discuss.elastic.co/t/comparing-two-fields-between-two-different-index-and-output-result-into-a-third-index/286535/2 "2021-10-12T19:10:00Z")

</div>

This kind of "joining" is probably best done in a relational database with an "index" (in SQL/rdbms speak) on the txn\_id. Then you can do a join on those two tables and columns and do whatever you need to do. This will be super quick!

If you REALLY want to use Elasticsearch, then take a look at this: [Joining queries | Elasticsearch Guide [7.15] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/joining-queries.html)  
However, these kinds of operations are incredibly expensive to do, so I don't recommend it in a high volume production environment.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2021, 7:10pm UTC](https://discuss.elastic.co/t/comparing-two-fields-between-two-different-index-and-output-result-into-a-third-index/286535/3 "2021-11-09T19:10:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.