Logstash - how to configure CSV filter for joining 2 CSV files based on a common field and send the joined log to ES

[Badger]

You are hitting the bug where the inode gets set to "unknown". So you have a sincedb file written by another instance of logstash that read a file called csv1. Luckily you do not want to be using a sincedb. This is wrong

sincedb_path => "null"

You will find there is a file called null somewhere. To suppress a persisent sincedb use

sincedb_path => "/dev/null"

on UNIX or

sincedb_path => "NUL"

on Windows.

[cheriemilk]

Thank you. lookup2.csv is created now. But there's no column header in it. file output filter doesn't keep the header when creating new file?

Regards,
Cherie

[Badger]

For the translate filter you do not need a header. But it should not have disappeared. You do not have a csv filter in the configuration that creates lookup2, right?

[cheriemilk]

Yes. in first logstash doesn't have CSV filter。

For dissect filter in second logstash, it will try to find header "lookup" in lookup2.csv which doesn't have a header created.

dissect { mapping => { "[@metadata][lookup]" => "%{FILE_CHANGED};%{FILE_TYPE};%{FILE_METHOD}" }}}

[Badger]

You do not need a header in lookup2.csv. Try it and you will see. You are never going to lookup the literal string "COMMIT_SHA1", so you do not need that value in the dictionary. You just need all the SHA1 hash strings in the dictionary.

[cheriemilk]

You're professional! Thanks a ton!

[Mahfoud_Gamraoui]

Hello Guys !!

I am in the same situation and i need your help here :
Am trying to merge 2 csv files based on a common field but unlike @cheriemilk i want in my output that i will parse in elasticsearch all the fields in both csv files:

in this case :

COMMIT_SHA1, FILE_CHANGED, FILE_TYPE, FILE_METHOD, GIT_ORG, GIT_REPOS, COMMIT_AUTHOR

Thanks in advance

[Mahfoud_Gamraoui]

Of course @Badger i've already read all your answers and it does not helped me becaus you are Pre-processing the second file so that it only has two columns.

[cheriemilk]

Not sure if I understand your expected correctly. Actually, The dissect filter in logstash config file will separated the merged columns(pre/processed ) from 1 to 3 after translate filter.