Multiple CSV files, Multiple Header data

Hi,
I am using multiple csv input files
with locations
/user/location2/consumption-*.csv
/user/location2/maps*.csv

Now in my logstash configuration i want to use conditional basis the path of csv files
input
{
file
{
path => [ "/user/location2/maps*.csv",
"/user/location2/consumption-*.csv"
]
start_position => "beginning"
# sincedb_path => "/user/somelocation/.sincedb"
}
}
filter {

if [path] == "/user/location2/maps*.csv"
{
csv {
        columns => [A,B,C,D]
        separator => ","
}
}
else if [path] == "/user/location2/consumption*.csv"
{
csv {
 columns => [A,E,F,G]
separator => ","
}
}

However I am not able to get the data pushed to elasticsearch

The path field will contain the actual path to the file, not the wildcard you used in the file input's path option. You can use a regular expression match conditional (see documentation) to check whether the path field matches a pattern.

But... I think it would be cleaner to split your single file input into two file inputs that set a tag or a particular field to indicate what kind of data it is. Then you can use a simple conditional on that tag or field to decide which filtes to apply.

Thanks Magnus,

So let me use this configuration input

input
{
file
{
path => "/user/location2/maps.csv"
start_position => "beginning"
#sincedb_path => "/user/somelocation/.sincedb"
}
file
 {
path => "/user/location2/consumption_data*.csv"
start_position => "beginning"
# sincedb_path => "/user/somelocation/.sincedb"
}
}
filter {
if [path] == "/user/location2/maps.csv"
{
csv {
        columns => [A,B,C,D]
        separator => ","
}
}
else if [path] == "/user/location2/consumption_data1.csv" , "/user/location2/consumption_data2.csv"
{
csv {
 columns => [A,E,F,G]
separator => ","
}
}

Will this work

else if [path] == "/user/location2/consumption_data1.csv" , "/user/location2/consumption_data2.csv"

This won't work, you need this:

else if [path] == "/user/location2/consumption_data1.csv" or [path] == "/user/location2/consumption_data2.csv"

Better yet, use a regular expression match.

But again, you should really be setting a tag or a field at the input.

1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.