# Logstash - how to configure CSV filter for joining 2 CSV files based on a common field and send the joined log to ES
**URL:** https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125
**Category:** Logstash
**Created:** [February 5, 2019, 1:08pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125 "2019-02-05T13:08:33Z")
**Posts on this page:** 10
**Page:** 2
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 7, 2019, 2:54pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/21 "2019-02-07T14:54:39Z")
You are hitting the bug where the inode gets set to "unknown". So you have a sincedb file written by another instance of logstash that read a file called csv1. Luckily you do not want to be using a sincedb. This is wrong
```
sincedb_path => "null"
```
You will find there is a file called null somewhere. To suppress a persisent sincedb use
```
sincedb_path => "/dev/null"
```
on UNIX or
```
sincedb_path => "NUL"
```
on Windows.
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [February 7, 2019, 11:07pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/22 "2019-02-07T23:07:44Z")
Thank you. lookup2.csv is created now. But there's no column header in it. file output filter doesn't keep the header when creating new file?
Regards,
Cherie
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 7, 2019, 11:11pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/23 "2019-02-07T23:11:23Z")
For the translate filter you do not need a header. But it should not have disappeared. You do not have a csv filter in the configuration that creates lookup2, right?
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [February 7, 2019, 11:12pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/24 "2019-02-07T23:12:00Z")
Yes. in first logstash doesn't have CSV filter。
For dissect filter in second logstash, it will try to find header "lookup" in lookup2.csv which doesn't have a header created.
dissect { mapping =\> { "[@metadata][lookup]" =\> "%{FILE\_CHANGED};%{FILE\_TYPE};%{FILE\_METHOD}" }}}
---
### Author:  [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [February 7, 2019, 11:22pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/25 "2019-02-07T23:22:17Z")
You do not need a header in lookup2.csv. Try it and you will see. You are never going to lookup the literal string "COMMIT\_SHA1", so you do not need that value in the dictionary. You just need all the SHA1 hash strings in the dictionary.
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [February 7, 2019, 11:46pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/26 "2019-02-07T23:46:01Z")
You're professional! Thanks a ton!
---
### Author:  [@Mahfoud\_Gamraoui](https://discuss.elastic.co/u/Mahfoud_Gamraoui)
#### Post date: [February 27, 2019, 10:11am UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/27 "2019-02-27T10:11:08Z")
Hello Guys !!
I am in the same situation and i need your help here :
Am trying to merge 2 csv files based on a common field but unlike @cheriemilk i want in my output that i will parse in elasticsearch all the fields in both csv files:
in this case :
**COMMIT\_SHA1** , FILE\_CHANGED, FILE\_TYPE, FILE\_METHOD, GIT\_ORG, GIT\_REPOS, COMMIT\_AUTHOR
Thanks in advance
---
### Author:  [@Mahfoud\_Gamraoui](https://discuss.elastic.co/u/Mahfoud_Gamraoui)
#### Post date: [February 27, 2019, 10:13am UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/28 "2019-02-27T10:13:00Z")
Of course @Badger i've already read all your answers and it does not helped me becaus you are Pre-processing the second file so that it only has two columns.
---
### Author:  [@cheriemilk](https://discuss.elastic.co/u/cheriemilk)
#### Post date: [February 28, 2019, 1:29pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/29 "2019-02-28T13:29:55Z")
Not sure if I understand your expected correctly. Actually, The dissect filter in logstash config file will separated the merged columns(pre/processed ) from 1 to 3 after translate filter.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 28, 2019, 1:29pm UTC](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125/30 "2019-03-28T13:29:56Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
[Previous page](https://discuss.elastic.co/t/logstash-how-to-configure-csv-filter-for-joining-2-csv-files-based-on-a-common-field-and-send-the-joined-log-to-es/167125.md?page=1)