Csv parse failure

Elastic Stack Logstash
1

Hello, I'm trying to parse a CSV file with Logstash, but I'm encountering a CSV parse failure. Can you please help me?

2

Hi Hanni,

Can you share the full error you are getting?

3 
, :exception=>#<CSV::MalformedCSVError: Unclosed quoted field in line 1.>}
[WARN ] 2023-05-11 10:38:31.439 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[WARN ] 2023-05-11 10:38:31.460 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"\",2023/05/09", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field in line 1.>}
[WARN ] 2023-05-11 10:38:31.463 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"11154,No,No,No,la,tcp,65,Unknown Service Detection: Banner Retrieval,There is an unknown service running on the remote host.,Nessus was unable to identify a service on the remote host even though it returned a banner of some type.,No,No,\"", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field in line 1.>}
[WARN ] 2023-05-11 10:38:31.474 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[WARN ] 2023-05-11 10:38:31.495 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"\",2023/05/09", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field in line 1.>}
[WARN ] 2023-05-11 10:38:31.498 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"11154,No,No,No,la,tcp,65,Unknown Service Detection: Banner Retrieval,There is an unknown service running on the remote host.,Nessus was unable to identify a service on the remote host even though it returned a banner of some type.,No,No,\"", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field in line 1.>}
[WARN ] 2023-05-11 10:38:31.507 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
[WARN ] 2023-05-11 10:38:31.531 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"\",2023/05/09", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field in line 1.>}
[WARN ] 2023-05-11 10:38:31.535 [[main]>worker0] csv - Error parsing csv {:field=>"message", :source=>"11154,No,No,No,la,tcp,65,Unknown Service Detection: Banner Retrieval,There is an unknown service running on the remote host.,Nessus was unable to identify a service on the remote host even though it returned a banner of some type.,No,No,\"", :exception=>#<CSV::MalformedCSVError: Unclosed quoted field in line 1.>}
[WARN ] 2023-05-11 10:39:09.720 [[main]>worker1] csv - Error parsing csv {:field=>"message", :source=>"", :exception=>#<NoMethodError: undefined method `each_index' for nil:NilClass>}
4

Can you check the format of your CSV? It looks like you might be missing a quote or an escaped character somewhere on the first line of your CSV file.

5

I checked it and didin't see any problem. I tried import my csv manually in elasticsearch and it worked fine. So I don't understand why it doesn't work with logstash

6

The line has a trailing double quote. That is not allowed.

1 Like
7

finally found that my file some data in his columns that Logstash is unable to parse. Is there a way to fix this?"

8

Are you able to use a mutate filter to change the value? I found an older topic that recommends that here.

9

I made this configuration

mutate {
     gsub => ["Solution", "[."]", " "]
 }

and I got this error

[ERROR] 2023-05-16 13:37:54.477 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \\t\\r\\n], \"#\", \"}\" at line 15, column 33 (byte 375) after filter {\n  csv {\n      separator => \",\"\n      skip_header => \"true\"\n      columns => [\"Plugin_ID\",\"CVE\",\"CVSS_v2.0_Base_Score\",\"Risk\",\"Host\",\"Protocol\",\"Port\",\"Solution\",\"date\"]\n  }\n  mutate {\n      gsub => [\"Solution\", \"[.\"]", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:239:in `initialize'", "org/logstash/execution/AbstractPipelineExt.java:173:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "org/jruby/RubyClass.java:911:in `new'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:50:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:386:in `block in converge_state'"]}
[INFO ] 2023-05-16 13:37:54.517 [LogStash::Runner] runner - Logstash shut down.
10

You cannot have an unquoted double quote inside double quotes. Perhaps gsub => ["Solution", '[."]', " "], but I am not sure what you are trying to do.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.