# Comparing fields of log files with different index in kibana **URL:** \ **Category:** Elasticsearch\ **Created:** [June 28, 2017, 12:41pm UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103 "2017-06-28T12:41:48Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Arpit](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Arpit](https://discuss.elastic.co/u/Arpit)\ **Post date:** [June 28, 2017, 12:41pm UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/1 "2017-06-28T12:41:48Z")
I have 2 different log files in elasticsearch, output as two different indexes I1 and I2 respectively in kibana. I want to compare the these two indexes or files and produce a result on the matching of field. Basically, for a row, if entry in field of I1 matches any entry form the entire file of I2 then that matched expression should be added as a field in I1 in that particular row. I am new to elasticsearch and environment and I tried doing it via DSL queries but couldn't find any luck. Please help me do that. Thank You ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 28, 2017, 1:25pm UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/2 "2017-06-28T13:25:25Z")
I don't think you can do that easily OOTB. May be you can scroll all the i1 index (fetch all documents) and one by on run a query on i2 to see if it matches but this is going to be slow for sure. You may be able to do that with Logstash with an elasticsearch input and elasticsearch filter. ---
**Author:** ![Arpit](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Arpit](https://discuss.elastic.co/u/Arpit)\ **Post date:** [July 3, 2017, 5:51am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/3 "2017-07-03T05:51:09Z")
I tried doing it by fetching one by one but then it becomes slow and does not serve the purpose. If you could tell how it can be done with logstash config file. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [July 3, 2017, 6:05am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/4 "2017-07-03T06:05:36Z")
If speed is an issue here, I don't think Logstash will solve it for this specific use case. Doing lookups is a slow task. ---
**Author:** ![Arpit](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Arpit](https://discuss.elastic.co/u/Arpit)\ **Post date:** [July 3, 2017, 6:20am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/5 "2017-07-03T06:20:55Z")
ok but I still need a way to do it with logstash and its filters ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 3, 2017, 6:32am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/6 "2017-07-03T06:32:15Z")
Can you tell us a bit more about the use case? What does the data in the 2 indices represent? How frequently are data in the index you are looking up against being added to or updated? What is the size of the index you are looking up against? Do you expect to always get a match? ---
**Author:** ![Arpit](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Arpit](https://discuss.elastic.co/u/Arpit)\ **Post date:** [July 3, 2017, 6:47am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/7 "2017-07-03T06:47:23Z")
Data in index1 is a log file which has field "DATA" and index2 has set of patterns to which I want to match the DATA filed in i1. Log files contain approximately 7-8K records and pattern file has in total of 14 patterns to be matched. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 3, 2017, 6:54am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/8 "2017-07-03T06:54:52Z")
Can you provide an example of the patterns and the log entries? If the index you are looking up against is quite static, you may be able to create a file based on the data and use the [translate filter plugin](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html), which supports strict key-value lookups as well as regular expressions. This would avoid the network round trips and might be more efficient. ---
**Author:** ![Arpit](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Arpit](https://discuss.elastic.co/u/Arpit)\ **Post date:** [July 3, 2017, 8:30am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/9 "2017-07-03T08:30:20Z")
The patterns and logs are confidential so I made some changes. So following provides the kind of patterns and log file field "DATA". Pattern : Error -103 creating case for service tag DATA : Response: \*\*\ ***, type=--** --, req#=register\_request, code=2, msg=Invalid value specified for SourceHeader.ClientHostName if any out of 14 patterns exist in any of the record in "DATA" field then it should show in I1 as another field for matched pattern. ---
**Author:** ![Arpit](https://avatars.discourse-cdn.com/v4/letter/a/e99b99/32.png) [@Arpit](https://discuss.elastic.co/u/Arpit)\ **Post date:** [July 4, 2017, 6:39am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/10 "2017-07-04T06:39:09Z")
Any help would make me closer to my project.. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 4, 2017, 7:10am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/11 "2017-07-04T07:10:38Z")
I do not understand the sample data you provided. Can you provide an example of each record type and describe how the match is done and what data you are looking to add to the processed event? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 1, 2017, 7:10am UTC](https://discuss.elastic.co/t/comparing-fields-of-log-files-with-different-index-in-kibana/91103/12 "2017-08-01T07:10:39Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.