Multiple CSV files, Multiple Header data

Elastic Stack Logstash
1

Hi,
I am using multiple csv input files
with locations
/user/location2/consumption-*.csv
/user/location2/maps*.csv

Now in my logstash configuration i want to use conditional basis the path of csv files
input
{
file
{
path => [ "/user/location2/maps*.csv",
"/user/location2/consumption-*.csv"
]
start_position => "beginning"
# sincedb_path => "/user/somelocation/.sincedb"
}
}
filter {

if [path] == "/user/location2/maps*.csv"
{
csv {
        columns => [A,B,C,D]
        separator => ","
}
}
else if [path] == "/user/location2/consumption*.csv"
{
csv {
 columns => [A,E,F,G]
separator => ","
}
}

However I am not able to get the data pushed to elasticsearch

2

The path field will contain the actual path to the file, not the wildcard you used in the file input's path option. You can use a regular expression match conditional (see documentation) to check whether the path field matches a pattern.

But... I think it would be cleaner to split your single file input into two file inputs that set a tag or a particular field to indicate what kind of data it is. Then you can use a simple conditional on that tag or field to decide which filtes to apply.

3

Thanks Magnus,

So let me use this configuration input

input
{
file
{
path => "/user/location2/maps.csv"
start_position => "beginning"
#sincedb_path => "/user/somelocation/.sincedb"
}
file
 {
path => "/user/location2/consumption_data*.csv"
start_position => "beginning"
# sincedb_path => "/user/somelocation/.sincedb"
}
}
filter {
if [path] == "/user/location2/maps.csv"
{
csv {
        columns => [A,B,C,D]
        separator => ","
}
}
else if [path] == "/user/location2/consumption_data1.csv" , "/user/location2/consumption_data2.csv"
{
csv {
 columns => [A,E,F,G]
separator => ","
}
}

Will this work

4 
else if [path] == "/user/location2/consumption_data1.csv" , "/user/location2/consumption_data2.csv"

This won't work, you need this:

else if [path] == "/user/location2/consumption_data1.csv" or [path] == "/user/location2/consumption_data2.csv"

Better yet, use a regular expression match.

But again, you should really be setting a tag or a field at the input.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.