How to use multiple csv files in logstash ,

Elastic Stack Logstash
1

I want to use multiple csv files in logstash hence please guide me . now using single csv file as per the below.

remaining file names are different ex : file1,file2,file3

input {
file {
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Region Name","Endpoint"]
}
}
output {
elasticsearch {
hosts => ""
index => "data-index"
}
stdout {}
}

2

Use can e.g. use different file inputs that each assign different types and then you can use conditionals to apply different filters and outputs to each kind of file.

https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html

3

Hi Magnus ,

Thanks alot for help,and not able to pass the values as per the below.

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
#sincedb_path => "/user/somelocation/.sincedb"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"

sincedb_path => "/user/somelocation/.sincedb"

}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => [A,B,C,D]
separator => ","
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv"
{
csv {
columns => [A,E,F,G]
separator => ","
}
}

4

So what happens? What does an example event look like when Logstash is done with it? Use a stdout { codec => rubydebug } output.

5

Hi Magnus ,
using the below and got the error

fetched an invalid config {:config=>"input\n{\nfile\n{\npath => "/tmp/AWSDiscoverynew.csv"\nstart_position => "beginning"\n}\nfile\n {\npath => "/tmp/AWSDiscoveryTest.csv"\nstart_position => "beginning"\n}\n}\nfilter {\nif [path] == "/tmp/AWSDiscoverynew.csv"\n{\ncsv {\n columns => ["Region Name","Endpoint"]\n separator => ","\n}\n}\nelse if [path] == "/tmp/AWSDiscoveryTest.csv" \n{\ncsv {\n columns => ["VpcId","State","CidrBlock","DhcpOptionsId","Tags","InstanceTenancy","IsDefault"]\nseparator => ","\n}\n}\noutput {\n elasticsearch {\n hosts => ""\n index => "data-index"\n }\nstdout { codec => rubydebug }\n}\n\n", :reason=>"Expected one of #, => at line 30, column 18 (byte 490) after filter {\nif [path] == "/tmp/AWSDiscoverynew.csv"\n{\ncsv {\n columns => ["Region Name","Endpoint"]\n separator => ","\n}\n}\nelse if [path] == "/tmp/AWSDiscoveryTest.csv" \n{\ncsv {\n columns => ["VpcId","State","CidrBlock","DhcpOptionsId","Tags","InstanceTenancy","IsDefault"]\nseparator => ","\n}\n}\noutput {\n elasticsearch ", :level=>:error}

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"
}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => ["Region Name","Endpoint"]
separator => ","
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv"
{
csv {
columns => ["VpcId","State","CidrBlock","DhcpOptionsId","Tags","InstanceTenancy","IsDefault"]
separator => ","
}
}
output {
elasticsearch {
hosts => ""
index => "data-index"
}
stdout { codec => rubydebug }

6

It looks like you're not closing the filter block before you begin your output block. Also, you're not closing your output block.

7

Please find the below config file and correct if anything wrong

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"
}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => ["A","B","C","D"]
separator => ","
}
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv" or [path] == "/tmp/AWSDiscoverytest2.csv"
}
{
csv {
columns => ["A","B","C","D","E","F","G"]
separator => ","
}
}
output {
elasticsearch {
hosts => "http://"
index => "data-index"
}
stdout {}

8

The hosts option in your elasticsearch output is obviously incorrect but otherwise it should work. However, having this kind of direct dependencies to filenames is a bad idea. Prefer setting the type or some other field in each file input and adjust your conditionals.

9

Thanks for the update,

I user the below log fie and got the below error,could you please check

The given configuration is invalid. Reason: Expected one of #, input, filter, output at line 28, column 1 (byte 351) after {:level=>:fatal}

input
{
file
{
path => "/tmp/AWSDiscoverynew.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoveryTest.csv"
start_position => "beginning"
}
file
{
path => "/tmp/AWSDiscoverytest2.csv"
start_position => "beginning"
}
}
filter {
if [path] == "/tmp/AWSDiscoverynew.csv"
{
csv {
columns => ["A","B","C","D"]
separator => ","
}
}
}
else if [path] == "/tmp/AWSDiscoveryTest.csv" or [path] == "/tmp/AWSDiscoverytest2.csv"
}
{
csv {
columns => ["A","B","C","D","E","F","G"]
separator => ","
}
}
output {
elasticsearch {
hosts => "http:localhost//"
index => "data-index"
}
stdout {}
}

10

You have too many braces before else if [path].

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.