CSV conf ig does not parse properly

Elastic Stack Logstash
1

I use a simple csv config file as shown below. A
input {
file {
path => "/Users/pgervais/Elk/mach.csv”
start_position => “beginning”
sincedb_path => “/dev/null”
}
}
filter {
csv {
convert => {
"ObsDate" => “date”
"ObsHour" => “integer”
"GPPLoadMIPS" => “float”
"zIIPLoadMIPS" => “float”
"GPPCapacityMIPS" => "float”
"zIIPCapacityMIPS" =>"float”
}
separator => “,”
autodetect_column_names => true
}
}

output {
elasticsearch {
hosts => "localhost"
index => "mach"
document_type => "mach_record"
}
stdout {}
}

According to csv filter docs, i should not get syntax errors on this. When i run it simply as a test using logstash-6.0.0/bin/logstash -f mach.conf -t i get the following error:
logstash-6.0.0/bin/logstash -f mach.conf -t
Sending Logstash's logs to /Users/pgervais/Elk/logstash-6.0.0/logs which is now configured via log4j2.properties
[2017-11-15T20:24:27,154][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/Users/pgervais/Elk/logstash-6.0.0/modules/fb_apache/configuration"}
[2017-11-15T20:24:27,161][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/Users/pgervais/Elk/logstash-6.0.0/modules/netflow/configuration"}
[2017-11-15T20:24:27,276][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2017-11-15T20:24:27,454][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of #, {, } at line 11, column 4 (byte 171) after input {
file {
path => "/Users/pgervais/Elk/mach.csv”
start_position => “beginning”
sincedb_path => “/dev/null”
}
}
filter {
csv {
convert => {
"

I also get errors when i try and specify the columns commnd:
columns => [ "Machine","ObsDate" , "ObsHour" , "GPPLoadMIPS", "zIIPLoadMIPS" , "GPPCapacityMIPS","zIIPCapacityMIPS" ]

It almost as if the csv is not loaded. So i checked via logstash-plugin logstash-6.0.0/bin/logstash-plugin list
The csv filter plugin is loaded.

Therefore why is the syntax as described in the logstash csv documentation generating errors?

I get this on version 5.6.4 and 6.00. i have followed the installation instructions.
What am i missing here?

2

"ObsDate" => “date”

Take a close look. It looks like you're mixing straight quotes with curly ones.

3

If you look at the documentation they are using " instead of '.
Will try ' and see what happens.

4

Yes, double quotes are used throughout the documentation but I wasn't talking about single vs. double quotes but straight vs. curly ones.

5

Magnus,
I appreciate your taking the time to reply but frankly i'm missing something here. What do mean you mean by "straight vs. curly one"?

I have changed my config file to use single quotes ' instead of double quotes. "
Most of the config file parses properly. It fails at the last line .

input {
file {
path => "/Users/pgervais/Elk/mach.csv”
start_position => “beginning”
sincedb_path => “/dev/null”
}
}
filter {
csv {
columns => ['Obs','Machine Serial#','Obs Date','Obs Hour','GPP Load (MIPS)','zIIP Load (MIPS)','GPP Capacity (MIPS)','zIIP Capacity (MIPS)' ]

    convert => { 
	'Obs Date' => date
    	'Obs Hour' => integer 
    	'GPP Load (MIPS)' =>  float
    	'zIIP Load (MIPS)' => float
    	'GPP Capacity (MIPS)' => float        
    	'zIIP Capacity (MIPS)' => float 
	}

Error shown below: I dont know what its looking for .

logstash-6.0.0/bin/logstash -f mach.conf -t
Sending Logstash's logs to /Users/pgervais/Elk/logstash-6.0.0/logs which is now configured via log4j2.properties
[2017-11-16T10:06:06,966][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/Users/pgervais/Elk/logstash-6.0.0/modules/fb_apache/configuration"}
[2017-11-16T10:06:06,980][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/Users/pgervais/Elk/logstash-6.0.0/modules/netflow/configuration"}
[2017-11-16T10:06:07,424][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2017-11-16T10:06:08,193][FATAL][logstash.runner ] The given configuration is invalid. Reason: Expected one of ", ", any character at line 34, column 1 (byte 687) after input {
file {
path => "/Users/pgervais/Elk/mach.csv”
start_position => “beginning”
sincedb_path => “/dev/null”
}
}
filter {
csv {
columns => ['Obs','Machine Serial#','Obs Date','Obs Hour','GPP Load (MIPS)','zIIP Load (MIPS)','GPP Capacity (MIPS)','zIIP Capacity (MIPS)' ]

    convert => { 
	'Obs Date' => date
    	'Obs Hour' => integer 
    	'GPP Load (MIPS)' =>  float
    	'zIIP Load (MIPS)' => float
    	'GPP Capacity (MIPS)' => float        
    	'zIIP Capacity (MIPS)' => float 
	}

separator => “,”

 }

}

output {
elasticsearch {
hosts => ‘localhost’
index => mach
document_type => mach_record
}
stdout {}
}

separator => “,”

 }

}

output {
elasticsearch {
hosts => ‘localhost’
index => mach
document_type => mach_record
}
stdout {}
}

6

I appreciate your taking the time to reply but frankly i'm missing something here. What do mean you mean by "straight vs. curly one"?

https://practicaltypography.com/straight-and-curly-quotes.html

Make sure all quotes in your configuration files are straight and not curly.

7

Magnus,
Can you take a look at the error im getting on my last post? I cant figure out what its expecting.

8

Nothing obvious, except that it still looks like curly quotes in several places. You should be able to comment out lines to narrow down where the error is. It could be a garbage character, possibly non-printable so it doesn't show up anywhere.

9

Marcus,
For all to know: The macbook air keyboard has a smart quote option. Even though i was typing the " key, it must have been doing a substitution. Something to keep in mind. I turned this option off and i can now use " in the config file.
Thanks for your help.

10

Marcus,
Ensured all " are of the proper type. Config file now parses properly.
Thanks for your help. Issue resolved.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.