Logstash plugin for reference data

[arunvijayam (Arun Prakash)]

Hi All,

I am reading 4 CSV files in csv input plugins with different type attribute with different headers. Is there any option/provision to merge multiple CSV files columns into single row of records in logstash.

Or

Is there any option/provision to refer another CSV content while doing ETL of main CSV file. For example, I am doing ETL for financial transactions to mongodb throw logstash from CSV input plugin. Then can I refer the reference data like USD value against Indian rupee on logstash filter plugin for money conversion. Can anybody help on this issue.

Thanks in Advance!

[magnusbaeck (Magnus Bäck)]

Have a look at the translate plugin.

[arunvijayam (Arun Prakash)]

Hi Magnusbaeuk,

Thanks for your idea,

Here is my problem my input code is as below

input {
file {
               type => "input1"
		path => ["C:\Official\Marsh\demo_csv_header.csv"]
		start_position => "beginning"
	}
file {
               type => "input2"
		path => ["C:\Official\Marsh\demo_csv_header.csv"]
		start_position => "beginning"
	}
}

Now my question is

1. how to refer input1 events in input2 events in filter
2. how to merge input1 events and input2 events in filter plugin and output plugin also.

Thanks in Advance!

[magnusbaeck (Magnus Bäck)]

Logstash doesn't support merging of events from different inputs (out of the box anyway, but a specialized plugin could help). As I said, have a look at the translate plugin.