Logstash makes new columns and gives different outputs on the same data

[ablaas]

Hi,
I am using logstash with the CSV plugin to put data in ES.

Problem is that logstash is generating extra columns while I don't see any extra fields in the CSV file.
Moreover, when I run logstash over the same csv input file, it generates a different output every time.

I run logstah from the CLI with a pipe : cat out.csv | logstash -config.conf
So data is send to the screen.

My data look like this:

......
192.168.1.13,62838,54.186.116.202,443,https,tcp,30188,2019-02-12 16:58:38.886959,sync-662-us-west-2.sync.services.mozilla.com,Uncategorized,tap2,FALSE
192.168.1.13,62848,172.217.17.42,443,https,tcp,7117,2019-02-12 17:05:45.567014,googleapis.l.google.com,Search_Engines/Portals,tap2,FALSE
192.168.1.13,61131,87.233.154.22,443,https,tcp,11878,2019-02-11 21:20:03.755531,media.amberalert.nl,Uncategorized,tap2,FALSE
1

Where the first record shown here (with 30188) gets columns added.

The logstash output for this record is:
"column17" => "Uncategorized",
"column16" => "sync-662-us-west-2.sync.services.mozilla.com",
"column15" => "2019-02-12 16:58:38.886959",
"column13" => "tcp",
"column14" => "30188",
"column18" => "tap2",
"column19" => "FALSE",
"column14" => "Uncategorized",
"column16" => "FALSE",

Running logstash again on the same data gives a complete different output.

csv processing is done by:
csv {
separator => ","
columns => [
"srcip",
"srcport",
"dstip",
"dstport",
"service",
"type",
"totalbytes",
"datum",
"hostname",
"category",
"tap",
"tornnet"
]
convert => {"port" => "integer"
"total bytes" => "integer"
}
} #csv

logstash version is 6.5.0

Hope someone can explain what's wrong.

[Badger]

By default, if you do not supply a column name for every column, then the csv filter will autogenerate them. If you do not want the additional columns parsed then set

autogenerate_column_names => false

As to why running the same command gives different results, you would need to give us a reproduceable test case. A complete configuration, test data, the command used, and details of what differs in the output.

[ablaas]

Thanks for the advice. Not generating extra columns will not solve it, because the data won't be placed in the right field.

I was looking for errors in my CSV file, but couldn't find any.

However, when looking at my .conf file I saw that it contained the twice the content file. (pasted it once to many). Removing the double the problem disappeared. Can't reproduce it anymore. It's strange that creating a wrong config file gives a non deterministic result.

However, thanks for looking at the problem