Logstash makes new columns and gives different outputs on the same data

I am using logstash with the CSV plugin to put data in ES.

Problem is that logstash is generating extra columns while I don't see any extra fields in the CSV file.
Moreover, when I run logstash over the same csv input file, it generates a different output every time.

I run logstah from the CLI with a pipe : cat out.csv | logstash -config.conf
So data is send to the screen.

My data look like this:

......,62838,,443,https,tcp,30188,2019-02-12 16:58:38.886959,sync-662-us-west-2.sync.services.mozilla.com,Uncategorized,tap2,FALSE,62848,,443,https,tcp,7117,2019-02-12 17:05:45.567014,googleapis.l.google.com,Search_Engines/Portals,tap2,FALSE,61131,,443,https,tcp,11878,2019-02-11 21:20:03.755531,media.amberalert.nl,Uncategorized,tap2,FALSE

Where the first record shown here (with 30188) gets columns added.

The logstash output for this record is:
"column17" => "Uncategorized",
"column16" => "sync-662-us-west-2.sync.services.mozilla.com",
"column15" => "2019-02-12 16:58:38.886959",
"column13" => "tcp",
"column14" => "30188",
"column18" => "tap2",
"column19" => "FALSE",
"column14" => "Uncategorized",
"column16" => "FALSE",

Running logstash again on the same data gives a complete different output.

csv processing is done by:
csv {
separator => ","
columns => [
convert => {"port" => "integer"
"total bytes" => "integer"
} #csv

logstash version is 6.5.0

Hope someone can explain what's wrong.

By default, if you do not supply a column name for every column, then the csv filter will autogenerate them. If you do not want the additional columns parsed then set

autogenerate_column_names => false

As to why running the same command gives different results, you would need to give us a reproduceable test case. A complete configuration, test data, the command used, and details of what differs in the output.

Thanks for the advice. Not generating extra columns will not solve it, because the data won't be placed in the right field.

I was looking for errors in my CSV file, but couldn't find any.

However, when looking at my .conf file I saw that it contained the twice the content file. (pasted it once to many). Removing the double the problem disappeared. Can't reproduce it anymore. It's strange that creating a wrong config file gives a non deterministic result.

However, thanks for looking at the problem

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.