# Matching data in two different indexes

**URL:** <https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741>\
**Category:** Kibana\
**Created:** [October 30, 2022, 1:55am UTC](https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741 "2022-10-30T01:55:32Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![OlegBochkarev](https://avatars.discourse-cdn.com/v4/letter/o/6a8cbe/32.png) [@OlegBochkarev](https://discuss.elastic.co/u/OlegBochkarev)\
**Post date:** [October 30, 2022, 1:55am UTC](https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741/1 "2022-10-30T01:55:32Z")

</div>

Hi, team!  
I collect authorising logs from two systems into two separate indexes. Index content:  
Index1:  
system1\_timestamp  
system1\_username  
system1\_ip  
...  
Index2:  
system2\_timesamp  
system2\_username  
system2\_ip  
...

I want find intersections on the following condition:  
User authorised one day with one ip in these two systems  
system1\_timestamp\_day == system2\_timestimestamp\_day AND system1\_ip == system2\_ip

On the output I want receive:  
system1\_username  
system2\_username  
system\_timestamp.day  
system\_ip

Now, to carry out this check, I use excel, which takes a lot of time(  
Tell me if it is possible to implement this matching by the Elastic tools?  
Thank you in advance!

---

<div class="post-metadata">

**Author:** ![Wolfram\_Haussig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wolfram_haussig/32/70528_2.png) [@Wolfram\_Haussig](https://discuss.elastic.co/u/Wolfram_Haussig)\
**Post date:** [October 31, 2022, 5:24am UTC](https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741/2 "2022-10-31T05:24:09Z")

</div>

Hello Oleg,

As far as I am aware joining two or more indexes is not possible in Kibana. Nevertheless, here are 2 options depending on your usecase:

**Option 1 - search for a specific IP only**  
Are the field names only placeholders for clarity or the real names? If those are the real names I would advise to synchronize the fieldnames between both indexes (e.g. by using [ECS](https://www.elastic.co/guide/en/ecs/current/ecs-user.html#field-user-name)):  
Index1:  
@timestamp  
user.name  
client.ip  
...  
Index2:  
@timestamp  
user.name  
client.ip  
...

After creating an [alias](https://www.elastic.co/guide/en/elasticsearch/reference/current/aliases.html) and creating an index pattern for this alias in Kibana, it is now possible to search for data regardless of the index the data is stored in. Searching for `client.ip: "127.0.0.1"` would return all documents from both Index1 and Index2 where this IP was mentioned.

**Option 2 - use Logstash to do what you want**  
To achieve exactly what you want, you could create a Logstash pipeline:

- use Elasticsearch input to search all data older than 1 hour but younger than 2 hours (to make sure to not prcoess documents twice but still making sure that the data for index2 is already stored in Elasticsearch)
- use Elasticsearch filter to enrich the document with data from Index2
- use drop filter to drop all documents where no matching document was found in index2
- use elasticsearch output to store the data in a new index

Now, you can search the new index exactly like you wanted.

I hope this helped pointing you in the right direction.

Best regards  
Wolfram

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 31, 2022, 11:59pm UTC](https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741/3 "2022-10-31T23:59:00Z")

</div>

Welcome to our community! 😃

Elasticsearch and Kibana cannot do joins dynamically like this. Another option is to run an ingest pipeline with an [enrich processor](https://www.elastic.co/guide/en/elasticsearch/reference/8.4/enrich-processor.html) to merge the two into a new index.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 28, 2022, 3:37am UTC](https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741/4 "2022-11-28T03:37:36Z")

</div>



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [December 26, 2022, 12:38am UTC](https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741/5 "2022-12-26T00:38:17Z")

</div>



---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 23, 2023, 12:39am UTC](https://discuss.elastic.co/t/matching-data-in-two-different-indexes/317741/6 "2023-01-23T00:39:16Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
