How to use multiple csv files in logstash ,

I want to use multiple csv files in logstash hence please guide me . now using single csv file as per the below.

remaining file names are different ex : file1,file2,file3

input {
file {
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Region Name","Endpoint"]
}
}
output {
elasticsearch {
hosts => ""
index => "data-index"
}
stdout {}
}

Use can e.g. use different file inputs that each assign different types and then you can use conditionals to apply different filters and outputs to each kind of file.

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

Hi Magnus ,

Thanks alot for help,and not able to pass the values as per the below.

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
#sincedb_path => "/user/somelocation/.sincedb"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"

sincedb_path => "/user/somelocation/.sincedb"

}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => [A,B,C,D]
separator => ","
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv"
{
csv {
columns => [A,E,F,G]
separator => ","
}
}

So what happens? What does an example event look like when Logstash is done with it? Use a stdout { codec => rubydebug } output.

Hi Magnus ,
using the below and got the error

fetched an invalid config {:config=>"input\n{\nfile\n{\npath => "/tmp/AWSDiscoverynew.csv"\nstart_position => "beginning"\n}\nfile\n {\npath => "/tmp/AWSDiscoveryTest.csv"\nstart_position => "beginning"\n}\n}\nfilter {\nif [path] == "/tmp/AWSDiscoverynew.csv"\n{\ncsv {\n columns => ["Region Name","Endpoint"]\n separator => ","\n}\n}\nelse if [path] == "/tmp/AWSDiscoveryTest.csv" \n{\ncsv {\n columns => ["VpcId","State","CidrBlock","DhcpOptionsId","Tags","InstanceTenancy","IsDefault"]\nseparator => ","\n}\n}\noutput {\n elasticsearch {\n hosts => ""\n index => "data-index"\n }\nstdout { codec => rubydebug }\n}\n\n", :reason=>"Expected one of #, => at line 30, column 18 (byte 490) after filter {\nif [path] == "/tmp/AWSDiscoverynew.csv"\n{\ncsv {\n columns => ["Region Name","Endpoint"]\n separator => ","\n}\n}\nelse if [path] == "/tmp/AWSDiscoveryTest.csv" \n{\ncsv {\n columns => ["VpcId","State","CidrBlock","DhcpOptionsId","Tags","InstanceTenancy","IsDefault"]\nseparator => ","\n}\n}\noutput {\n elasticsearch ", :level=>:error}

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"
}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => ["Region Name","Endpoint"]
separator => ","
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv"
{
csv {
columns => ["VpcId","State","CidrBlock","DhcpOptionsId","Tags","InstanceTenancy","IsDefault"]
separator => ","
}
}
output {
elasticsearch {
hosts => ""
index => "data-index"
}
stdout { codec => rubydebug }

It looks like you're not closing the filter block before you begin your output block. Also, you're not closing your output block.

Please find the below config file and correct if anything wrong

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"
}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => ["A","B","C","D"]
separator => ","
}
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv" or [path] == "/tmp/AWSDiscoverytest2.csv"
}
{
csv {
columns => ["A","B","C","D","E","F","G"]
separator => ","
}
}
output {
elasticsearch {
hosts => "http://"
index => "data-index"
}
stdout {}

The hosts option in your elasticsearch output is obviously incorrect but otherwise it should work. However, having this kind of direct dependencies to filenames is a bad idea. Prefer setting the type or some other field in each file input and adjust your conditionals.

Thanks for the update,

I user the below log fie and got the below error,could you please check

The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 28, column 1 (byte 351) after {:level=>:fatal}

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoverytest2.csv"
start_position => "beginning"
}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => ["A","B","C","D"]
separator => ","
}
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv" or [path] == "/tmp/AWSDiscoverytest2.csv"
}
{
csv {
columns => ["A","B","C","D","E","F","G"]
separator => ","
}
}
output {
elasticsearch {
hosts => "http:localhost//"
index => "data-index"
}
stdout {}
}

You have too many braces before else if [path].

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.