# CSV filter duplicate output result **URL:** \ **Category:** Logstash\ **Created:** [January 25, 2020, 1:35pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521 "2020-01-25T13:35:28Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![pierrejutard](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@pierrejutard](https://discuss.elastic.co/u/pierrejutard)\ **Post date:** [January 25, 2020, 1:35pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/1 "2020-01-25T13:35:28Z")
Hello, Under windows 10, with 7.5.2 ES stack version, I try to make a pipeline from a csv file (";") delimiter to ES but the output is not stable. For instance it duplicates the result: The output is:{ "twod" =\> 0.0, "oned" =\> 0.0, "@timestamp" =\> 2020-01-25T13:21:09.117Z, "spot" =\> 0.0 } { "twod" =\> 6.0, "oned" =\> 3.0, "@timestamp" =\> 2020-01-25T13:21:09.138Z, "spot" =\> 2.0 } With the following config file: input { file { path =\> "C:/Users/jutar/OneDrive/Desktop/mktdata.csv" start\_position =\> "beginning" sincedb\_path =\> "NUL" } } filter { csv { separator =\> ";" columns =\> ["spot","oned","twod"] remove\_field =\> ["host","path","@version","message"] } mutate{convert =\>["spot","float"]} mutate{convert =\>["oned","float"]} mutate{convert =\>["twod","float"]} } output { elasticsearch { hosts =\> "[http://localhost:9200](http://localhost:9200)" index =\> "mktdata-%{+YYYY.MM.dd}" } stdout {} } And when using these lines in the config file csv filter: csv { autodetect\_column\_names =\> true } , it sometimes inverts the oupt column, i.e.: ``` "6" => twod, "3" => oned, "@timestamp" => 2020-01-25T13:21:09.138Z, "2" => spot ``` My csv file is the following: "spot" in A1 cell "oned" in B1 and "twod" in C1 "2" in A2 cell "3" in in B2 cell and "6" in C2 cell. Can anyone know the answer of this strange parsing? Thank you in advance Pierre Jutard ---
**Author:** ![pierrejutard](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@pierrejutard](https://discuss.elastic.co/u/pierrejutard)\ **Post date:** [January 25, 2020, 1:52pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/2 "2020-01-25T13:52:10Z")
This is how Logstash see the message i have to parse: { "column1" =\> "?spot;oned;twod", "message" =\> "?spot;oned;twod\r" } { "column1" =\> "2;3;6", "message" =\> "2;3;6\r" } ---
**Author:** ![pierrejutard](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@pierrejutard](https://discuss.elastic.co/u/pierrejutard)\ **Post date:** [January 25, 2020, 2:52pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/3 "2020-01-25T14:52:37Z")
Hello guys, After investigation, when using this filter: filter { csv { autodetect\_column\_names =\> true separator =\> ";" remove\_field =\> ["host","path","@version","message"] } When saving with this extension CSV (DOS): it inverts the column with their respective values: { "5" =\> "oned", "7" =\> "twod", "@timestamp" =\> 2020-01-25T14:47:01.840Z, "3" =\> "spot" } but when saving with this extension CSV ( semi-colon) it is working: { "twod" =\> 7.0, "oned" =\> 5.0, "@timestamp" =\> 2020-01-25T14:46:02.416Z, "spot" =\> 3.0 } Pierre Jutard ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 25, 2020, 3:51pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/4 "2020-01-25T15:51:22Z")
When using autodetect\_column\_names you must set pipeline.workers to 1, and also disable [java\_execution](https://github.com/elastic/logstash/issues/10938). ---
**Author:** ![pierrejutard](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@pierrejutard](https://discuss.elastic.co/u/pierrejutard)\ **Post date:** [January 25, 2020, 7:33pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/5 "2020-01-25T19:33:15Z")
Ok i already used the default pipeline.workers setting.(=1 apparently) and the java execution should be set to false when launching the pipelines in Logstash like this: bin/logstash --java-execution=false but is there another simple way to do it? Pierre Jutard ---
**Author:** ![ropc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ropc/32/47022_2.png) [@ropc](https://discuss.elastic.co/u/ropc)\ **Post date:** [January 26, 2020, 3:13am UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/6 "2020-01-26T03:13:02Z")
@pierrejutard - You can use one of the following option: - [command-line flag](https://www.elastic.co/guide/en/logstash/7.5/running-logstash-command-line.html) `--java-execution` and set the value to `false`. - `pipeline.java_execution` [parameter](https://www.elastic.co/guide/en/logstash/7.5/logstash-settings-file.html) and set the value to `false` in the `logstash.yml` file. ---
**Author:** ![pierrejutard](https://avatars.discourse-cdn.com/v4/letter/p/c5a1d2/32.png) [@pierrejutard](https://discuss.elastic.co/u/pierrejutard)\ **Post date:** [January 27, 2020, 5:59pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/7 "2020-01-27T17:59:58Z")
Thank you Romain it worked and it's a simple way. Best, Pierre Jutard ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 24, 2020, 6:00pm UTC](https://discuss.elastic.co/t/csv-filter-duplicate-output-result/216521/8 "2020-02-24T18:00:05Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.