# Logstash makes new columns and gives different outputs on the same data

**URL:** <https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411>\
**Category:** Logstash\
**Created:** [February 14, 2019, 12:22pm UTC](https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411 "2019-02-14T12:22:22Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ablaas](https://avatars.discourse-cdn.com/v4/letter/a/e95f7d/32.png) [@ablaas](https://discuss.elastic.co/u/ablaas)\
**Post date:** [February 14, 2019, 12:22pm UTC](https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411/1 "2019-02-14T12:22:23Z")

</div>

Hi,  
I am using logstash with the CSV plugin to put data in ES.

Problem is that logstash is generating extra columns while I don't see any extra fields in the CSV file.  
Moreover, when I run logstash over the same csv input file, it generates a different output every time.

I run logstah from the CLI with a pipe : cat out.csv | logstash -config.conf  
So data is send to the screen.

My data look like this:

......  
192.168.1.13,62838,54.186.116.202,443,https,tcp,30188,2019-02-12 16:58:38.886959,[sync-662-us-west-2.sync.services.mozilla.com](http://sync-662-us-west-2.sync.services.mozilla.com),Uncategorized,tap2,FALSE  
192.168.1.13,62848,172.217.17.42,443,https,tcp,7117,2019-02-12 17:05:45.567014,[googleapis.l.google.com](http://googleapis.l.google.com),Search\_Engines/Portals,tap2,FALSE  
192.168.1.13,61131,87.233.154.22,443,https,tcp,11878,2019-02-11 21:20:03.755531,media.amberalert.nl,Uncategorized,tap2,FALSE  
1

Where the first record shown here (with 30188) gets columns added.

The logstash output for this record is:  
"column17" =\> "Uncategorized",  
"column16" =\> "[sync-662-us-west-2.sync.services.mozilla.com](http://sync-662-us-west-2.sync.services.mozilla.com)",  
"column15" =\> "2019-02-12 16:58:38.886959",  
"column13" =\> "tcp",  
"column14" =\> "30188",  
"column18" =\> "tap2",  
"column19" =\> "FALSE",  
"column14" =\> "Uncategorized",  
"column16" =\> "FALSE",

Running logstash again on the same data gives a complete different output.

csv processing is done by:  
csv {  
separator =\> ","  
columns =\> [  
"srcip",  
"srcport",  
"dstip",  
"dstport",  
"service",  
"type",  
"totalbytes",  
"datum",  
"hostname",  
"category",  
"tap",  
"tornnet"  
]  
convert =\> {"port" =\> "integer"  
"total bytes" =\> "integer"  
}  
} #csv

logstash version is 6.5.0

Hope someone can explain what's wrong.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 14, 2019, 2:23pm UTC](https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411/2 "2019-02-14T14:23:08Z")

</div>

By default, if you do not supply a column name for every column, then the csv filter will autogenerate them. If you do not want the additional columns parsed then set

```
autogenerate_column_names => false

```

As to why running the same command gives different results, you would need to give us a reproduceable test case. A complete configuration, test data, the command used, and details of what differs in the output.

---

<div class="post-metadata">

**Author:** ![ablaas](https://avatars.discourse-cdn.com/v4/letter/a/e95f7d/32.png) [@ablaas](https://discuss.elastic.co/u/ablaas)\
**Post date:** [February 14, 2019, 4:58pm UTC](https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411/3 "2019-02-14T16:58:57Z")

</div>

Thanks for the advice. Not generating extra columns will not solve it, because the data won't be placed in the right field.

I was looking for errors in my CSV file, but couldn't find any.

However, when looking at my .conf file I saw that it contained the twice the content file. (pasted it once to many). Removing the double the problem disappeared. Can't reproduce it anymore. It's strange that creating a wrong config file gives a non deterministic result.

However, thanks for looking at the problem

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 14, 2019, 4:58pm UTC](https://discuss.elastic.co/t/logstash-makes-new-columns-and-gives-different-outputs-on-the-same-data/168411/4 "2019-03-14T16:58:58Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.