I am building a conf file to use a template for all other csv ingestion to ES indexes I need. This conf was working (in a simpler) form and is based on a v2 dev logstash install. Question...
Has the syntax changed when using file filter and csv plug-in?
Should column names always be lowercase only?
Logstash says my output is the problem. I am using Sublime 3 on a mac and starting to suspect my editor is creating characters unfriendly for logstash. What text editor is safe to use for conf files on osx?
btw: Line 40 Column 18 is the line...elasticsearch {
Here is my error message and conf file
greg$ bin/logstash -f ~/logstash-5.2.1/test_file.conf --debug --verbose
ERROR StatusLogger No log4j2 configuration file found. Using default configuration: logging only errors to the console.
Sending Logstash's logs to /Users/greg/logstash-5.2.1/logs which is now configured via log4j2.properties
[2017-03-15T21:43:22,699][ERROR][logstash.agent ] fetched an invalid config {:config=>"input {\nfile {\npath => "/Users/greg/logstash-5.2.1/efast_master.csv"\nstart_position => "beginning"\nsincedb_path => "/dev/null"\n}\n}\nfilter {\ncsv {\nseparator => ","\ncolumns => ["Company","MOA_Number","MOA_Org","TaskOrder","ProjectName",\n "AwardType","FunctionalArea","AwardDate","EndDate","OptionsRemaining",\n "TotalValue","PMORouteSymbol"]\n}\nmutate {\n \n convert => { \n "TotalValue" => "float" \n "TaskOrder" => "float" \n \n }\n }\n\ndate {\n match => [ "AwardDate" , "MM/dd/yyyy" ,"M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy"]\n }\n date {\n match => [ "AwardDate" , "MM/dd/yyyy","M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy" ]\n target => "AwardDate"\n }\ndate {\n match => [ "EndDate" , "MM/dd/yyyy" ,"M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy"]\n }\n date {\n match => [ "EndDate" , "MM/dd/yyyy","M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy" ]\n target => "EndDate"\n }\n\noutput {\n elasticsearch {\n hosts => "http://localhost:9200"\n index => "efast"\n }\nstdout { codec => rubydebug }\n}\n", :reason=>"Expected one of #, => at line 40, column 19 (byte 953) after filter {\ncsv {\nseparator => ","\ncolumns => ["Company","MOA_Number","MOA_Org","TaskOrder","ProjectName",\n "AwardType","FunctionalArea","AwardDate","EndDate","OptionsRemaining",\n "TotalValue","PMORouteSymbol"]\n}\nmutate {\n \n convert => { \n "TotalValue" => "float" \n "TaskOrder" => "float" \n \n }\n }\n\ndate {\n match => [ "AwardDate" , "MM/dd/yyyy" ,"M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy"]\n }\n date {\n match => [ "AwardDate" , "MM/dd/yyyy","M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy" ]\n target => "AwardDate"\n }\ndate {\n match => [ "EndDate" , "MM/dd/yyyy" ,"M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy"]\n }\n date {\n match => [ "EndDate" , "MM/dd/yyyy","M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy" ]\n target => "EndDate"\n }\n\noutput {\n elasticsearch "}
[CONF FILE BELOW]
input {
file {
path => "/Users/greg/logstash-5.2.1/efast_master.csv"
start_position => "beginning"
sincedb_path => "/dev/null"
}
}
filter {
csv {
separator => ","
columns => ["Company","MOA_Number","MOA_Org","TaskOrder","ProjectName",
"AwardType","FunctionalArea","AwardDate","EndDate","OptionsRemaining",
"TotalValue","PMORouteSymbol"]
}
mutate {
convert => {
"TotalValue" => "float"
"TaskOrder" => "float"
}
}
date {
match => [ "AwardDate" , "MM/dd/yyyy" ,"M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy"]
}
date {
match => [ "AwardDate" , "MM/dd/yyyy","M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy" ]
target => "AwardDate"
}
date {
match => [ "EndDate" , "MM/dd/yyyy" ,"M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy"]
}
date {
match => [ "EndDate" , "MM/dd/yyyy","M/dd/yyyy","MM/d/yyyy" ,"M/d/yyyy" ]
target => "EndDate"
}
output {
elasticsearch {
hosts => "http://localhost:9200"
index => "efast"
}
stdout { codec => rubydebug }
}