Logstash - how to configure CSV filter for joining 2 CSV files based on a common field and send the joined log to ES

You are hitting the bug where the inode gets set to "unknown". So you have a sincedb file written by another instance of logstash that read a file called csv1. Luckily you do not want to be using a sincedb. This is wrong

sincedb_path => "null"

You will find there is a file called null somewhere. To suppress a persisent sincedb use

sincedb_path => "/dev/null"

on UNIX or

sincedb_path => "NUL"

on Windows.

Thank you. lookup2.csv is created now. But there's no column header in it. file output filter doesn't keep the header when creating new file?

Regards,
Cherie

For the translate filter you do not need a header. But it should not have disappeared. You do not have a csv filter in the configuration that creates lookup2, right?

Yes. in first logstash doesn't have CSV filter。

For dissect filter in second logstash, it will try to find header "lookup" in lookup2.csv which doesn't have a header created.

dissect { mapping => { "[@metadata][lookup]" => "%{FILE_CHANGED};%{FILE_TYPE};%{FILE_METHOD}" }}}

You do not need a header in lookup2.csv. Try it and you will see. You are never going to lookup the literal string "COMMIT_SHA1", so you do not need that value in the dictionary. You just need all the SHA1 hash strings in the dictionary.

1 Like

You're professional! Thanks a ton!

Hello Guys !!

I am in the same situation and i need your help here :
Am trying to merge 2 csv files based on a common field but unlike @cheriemilk i want in my output that i will parse in elasticsearch all the fields in both csv files:

in this case :

COMMIT_SHA1, FILE_CHANGED, FILE_TYPE, FILE_METHOD, GIT_ORG, GIT_REPOS, COMMIT_AUTHOR

Thanks in advance

Of course @Badger i've already read all your answers and it does not helped me becaus you are Pre-processing the second file so that it only has two columns.

Not sure if I understand your expected correctly. Actually, The dissect filter in logstash config file will separated the merged columns(pre/processed ) from 1 to 3 after translate filter.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.