Hi all, I am seeing WARNING messages when csv is pushed to logstash, I want to eliminate the WARNING messages, any help appreciated:
Sample csv:
20185656,2021-02-01 17:52:47,2021-02-01 18:23:15,"Hi Test2, this is check.com/L",ee53b46d906232b0f925b456,,"",+12345445,tail,44a87564-a85f-4aa8-9e82f,Unclassified,"None","None",""
conf file:
input {
file {
path => "/var/lib/logstash/test/*.csv"
start_position => "beginning"
add_field => { "type" => "test" }
}
}
filter {
csv {
columns => ["id","time","updatetime","text","another","number","number2","number3","source","number4","class","check","check1","check2"]
separator => ","
}
date {
match => ["reporttime","yyyy-MM-dd HH:mm:ss"]
}
}
output {
elasticsearch {
hosts => ["XXX.XXX.XXX.X:9200"]
index => "test1"
}
}
Error seen while tailing logstash-plain.log:
[2021-02-11T19:31:04,910][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2021-02-11T19:32:05,287][WARN ][logstash.filters.csv ][main][48514e46dc7d34714c27b3b371a9ed647d2e2aee6edc72dc7b32e3364adc1822] Error parsing csv {:field=>"message", :source=>"20185656,2021-02-01 17:52:47,2021-02-01 18:23:15,\"Hi Test2, this is check.com/L\",ee53b46d906232b0f925b456,,\"\",+12345445,tail,44a87564-a85f-4aa8-9e82f,Unclassified,\"None\",\"None\",\"\"", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
[2021-02-11T19:32:05,301][WARN ][logstash.filters.csv ][main][aaf4724aa89d5ab109e193a34dd6d144e9f4b45a268932f3c90bb3e70deede4a] Error parsing csv {:field=>"message", :source=>"20185656,2021-02-01 17:52:47,2021-02-01 18:23:15,\"Hi Test2, this is check.com/L\",ee53b46d906232b0f925b456,,\"\",+12345445,tail,44a87564-a85f-4aa8-9e82f,Unclassified,\"None\",\"None\",\"\"", :exception=>#<CSV::MalformedCSVError: Illegal quoting in line 1.>}
I have tested removing all the quotations from the message and with this no _csvparsefailure warning is seen in tags. However I want to be able to keep the quotations as is, in fields they are present in and see no _csvparsefailure message. Is this possible?
In my case there does not appear to be any encoding error or issue with quote_char as I have seen in other posts.