Parsing CSV => columns semicolon separated => values with commas

[jiona]

Hi all,

I am trying to parse a csv files with logstash.
The csv contains semicolon separated columns and values contain commas.

That's my csv-file:

Study number;Series description;Value1;Value2;Value3
5870;Text Information;3221;0,09;
5870;Text Information;3221;0,09;
5870;Text Information;15650;13,29;
5850;Text Information;3560;0,09;
5850;Text Information;6495;2,89;2,89
5850;Text Information;4593;2,49;2,49

Thats my logstash filter section:

filter {
csv {
separator => ";"
columns => ["Study number";"Series description";"Value1";"Value2";"Value3"]
}
}

After stating the import I get following error message:

[2018-05-13T12:14:44,895][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, ,, ] at line 11, column 32 (byte 198) after filter {\n csv {\n separator => ";"\n columns => ["Study number"", :backtrace=>["/usr/............

I read some articles and understand that's a problem of double quotes but I found no real explanation and solution .
Can someone help?

Kind regards - jiona

[Christian_Dahlqvist]

@jiona

columns => ["Study number";"Series description";"Value1";"Value2";"Value3"]

This should be separated by commas, not semi-colons.

[jiona]

Hi Christian.
I will try immediately.
Give me 5 min.

[jiona]

Great - it works.

That's my simple filter:

filter {
csv {
separator => ";"
columns => ["Study number","Series description","Value1","Value2","Value3"]
}
}

Thats the output of "codec => rubydebug"

{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5870",
"@timestamp" => 2018-05-13T10:52:02.988Z,
"Series description" => "Text Information",
"Value2" => "0,09",
"message" => "5870;Text Information;3221;0,09;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "3221"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5870",
"@timestamp" => 2018-05-13T10:52:02.989Z,
"Series description" => "Text Information",
"Value2" => "13,29",
"message" => "5870;Text Information;15650;13,29;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "15650"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5842",
"@timestamp" => 2018-05-13T10:52:02.992Z,
"Series description" => "Text Information",
"Value2" => "0,09",
"message" => "5842;Text Information;1661;0,09;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "1661"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5842",
"@timestamp" => 2018-05-13T10:52:02.993Z,
"Series description" => "Text Information",
"Value2" => "6,9",
"message" => "5842;Text Information;11362;6,9;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "11362"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5850",
"@timestamp" => 2018-05-13T10:52:02.994Z,
"Series description" => "Text Information",
"Value2" => "2,89",
"message" => "5850;Text Information;6495;2,89;2,89",
"Value3" => "2,89",
"host" => "1001SITLOG11",
"Value1" => "6495"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "Study number",
"@timestamp" => 2018-05-13T10:52:02.986Z,
"Series description" => "Series description",
"Value2" => "Value2",
"message" => "Study number;Series description;Value1;Value2;Value3",
"Value3" => "Value3",
"host" => "1001SITLOG11",
"Value1" => "Value1"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5870",
"@timestamp" => 2018-05-13T10:52:02.988Z,
"Series description" => "Text Information",
"Value2" => "0,09",
"message" => "5870;Text Information;3221;0,09;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "3221"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5843",
"@timestamp" => 2018-05-13T10:52:02.990Z,
"Series description" => "Text Information",
"Value2" => "0,18",
"message" => "5843;Text Information;3060;0,18;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "3060"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5842",
"@timestamp" => 2018-05-13T10:52:02.993Z,
"Series description" => "Text Information",
"Value2" => "0,09",
"message" => "5842;Text Information;1661;0,09;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "1661"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5850",
"@timestamp" => 2018-05-13T10:52:02.993Z,
"Series description" => "Text Information",
"Value2" => "0,09",
"message" => "5850;Text Information;3560;0,09;",
"Value3" => nil,
"host" => "1001SITLOG11",
"Value1" => "3560"
}
{
"path" => "/home/portal/Portal.csv",
"@version" => "1",
"Study number" => "5850",
"@timestamp" => 2018-05-13T10:52:02.994Z,
"Series description" => "Text Information",
"Value2" => "2,49",
"message" => "5850;Text Information;4593;2,49;2,49",
"Value3" => "2,49",
"host" => "1001SITLOG11",
"Value1" => "4593"
}

Best regards - muchas gracias