# Error parsing csv file // logstash

**URL:** <https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963>\
**Category:** Logstash\
**Created:** [December 16, 2019, 9:02am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963 "2019-12-16T09:02:05Z")\
**Posts on this page:** 14\
**Page:** 1

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 16, 2019, 9:02am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/1 "2019-12-16T09:02:05Z")

</div>

Hi,

I'm checking my logstash logs and see many entries with WARN "Error parsing csv" (80% cases are well parsed). ¿could u help me out? Many thanks in advance.

Furthermore, "date" field is not added to timestamp index.

Logstash conf file:

input {  
file {  
path =\> "/home/adminfran/desktop/monitoring/act\_final3.csv"  
start\_position =\> "beginning"  
sincedb\_path =\> "/dev/null"  
}  
}

```
    filter {
        csv {
			      columns => ["id tweet","date","author","text","app","id user","followers","following","stauses","location","urls","geolocation","name","description","url_media","type media","quoted","relation","replied_id","user replied","retweeted_id","user retweeted","quoted_id","user quoted","first HT","lang","created_at","verified","avatar","link"]
			      separator => "	" #tab
			    }
		      date {
          match => ["date","yyyy-MM-dd HH:mm:ss"]
          timezone => "UTC"
          target => "date"
		      }
        mutate {
          remove_field => ["message"]
        }
	    }

```

output {  
elasticsearch {  
hosts =\> ["[http://localhost:9200](http://localhost:9200)"]  
index =\> "monitoring\_twitter"  
}  
stdout{codec =\> rubydebug}  
}

Error:  
[WARN] 2019-12-13 13:03:17.881 [[main]\>worker1] csv - Error parsing csv {:field=\>"message", :source=\>"1146480753682898944\t2019-07-03 18:08:06\t@monterhusmore\tRT @Sertemus: Sagen sie mir, wenn ich falsch liege: hatten polizeibeamte, die @KRLS beobachteten, bereits einen gerichtsbeschluss? Wenn nicht, wer erteilt ihnen den auftrag, diese überwachung auf deutschem gebiet durchzuführen? @DerSPIEGEL @nytimes @guardian @BILD @lalsace @washingtonpost [https://t.co/XsVtCC0FQi\tTwitter](https://t.co/XsVtCC0FQi%5CtTwitter) for Android\t2761881400\t2280\t4979\t200492\tNone\thttps://twitter.com/portet\_bruguera/status/1146069575462637570\tNone\tMONTERHUSMORE.\tNone\tNone\tNone\tMHP @KRLS "El TJUE ha de resoldre, ha d'intervenir. Alguns ens voldrien silenciats, quiets i tancats. No hem callat, no ens aturarem i no ens rendirem. Deixeu-me dir ben clar: visca Europa i visca Catalunya Lliure" #PersisitimiGuanyarem [https://t.co/5HsnXf6pGO\tquote\tNone\tNone\tNone\tNone\t1146069575462637570\tportet\_bruguera\tNone\tde\t2014-09-05](https://t.co/5HsnXf6pGO%5Ctquote%5CtNone%5CtNone%5CtNone%5CtNone%5Ct1146069575462637570%5Ctportet_bruguera%5CtNone%5Ctde%5Ct2014-09-05) 10:39:56\tFalse\thttps://pbs.twimg.com/profile\_images/1139420260917006336/dUbIjzwm\_normal.jpg\thttps://twitter.com/monterhusmore/status/1146480753682898944", :exception=\>#\<CSV::MalformedCSVError: Illegal quoting in line 1.\>}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [December 16, 2019, 3:38pm UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/2 "2019-12-16T15:38:12Z")

</div>

> [@frangolzmil](#):
>
> \tMHP @KRLS "El TJUE

In a CSV file the entire field has to be quoted, and quotes within it are represented using double double quotes. You cannot have part of a field quoted.

---

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 17, 2019, 9:24am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/3 "2019-12-17T09:24:37Z")

</div>

So, I'm trying to transform field values in order to get rid of this character however I do not know to do it  
I tried this:  
mutate {  
gsub =\> ["message", """, "''"]  
remove\_field =\> ["message"]  
}  
But It is not working. ¿Could you help me?

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [December 17, 2019, 9:28am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/4 "2019-12-17T09:28:01Z")

</div>

Hi

You have to escape the characters:

```auto
gsub => ["message","\"","\'"]

```

(maybe not the "'", give it a try)

and you should probably not remove "message" at this stage.

Hope this helps

---

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 17, 2019, 9:37am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/5 "2019-12-17T09:37:13Z")

</div>

> [@ITIC](#):
>
> gsub =\> ["message",""","'"]

That is happening on visualcode:

 ![imagen](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b2f938ef516668e0a60e2bb73d218958fef1d7d9.png)

---

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 17, 2019, 9:40am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/6 "2019-12-17T09:40:47Z")

</div>

ERROR:

[ERROR] 2019-12-17 10:38:47.948 [Converge PipelineAction::Create] agent - Failed to execute action {:action=\>LogStash::PipelineAction::Create/pipeline\_id:main, :exception=\>"LogStash::ConfigurationError", :message=\>"Expected one of [\t\r\n], "#", "{", ",", "]" at line 21, column 39 (byte 855) after filter {\n csv {\n\t\t\t\t columns =\> ["id tweet","date","author","text","app","id user","followers","following","stauses","location","urls","geolocation","name","description","url\_media","type media","quoted","relation","replied\_id","user replied","retweeted\_id","user retweeted","quoted\_id","user quoted","first HT","lang","created\_at","verified","avatar","link"]\n\t\t\t\t separator =\> "\t" #tabulaciones\n\t\t\t\t }\n\t\t\t date {\n match =\> ["date","yyyy-MM-dd HH:mm:ss"]\n timezone =\> "UTC"\n target =\> "@timestamp"\n\t\t\t }\n \n mutate {\n gsub =\> ["message", "\""", :backtrace=\>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:41:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:49:in `compile\_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `block in compile_sources'", "org/jruby/RubyArray.java:2584:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:10:in `compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:156:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:47:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:26:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline\_action/create.rb:36:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:326:in `block in converge\_state'"]}

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [December 17, 2019, 9:43am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/7 "2019-12-17T09:43:09Z")

</div>

Hi

Try without escaping the single quote:

```auto
gsub => ["message","\"","'"]

```

Sometimes comments screw up. Try removing the "remove\_field" line alltoghether.

---

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 17, 2019, 10:09am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/9 "2019-12-17T10:09:14Z")

</div>

It is not working

 ![imagen](https://us1.discourse-cdn.com/elastic/original/3X/e/a/eabe602fb421aa2186d56a8cff98d5038cf1e058.png)

---

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 17, 2019, 10:14am UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/10 "2019-12-17T10:14:13Z")

</div>

Sorry, It is working, but not well parsed again. Look this pic:

 ![imagen](https://us1.discourse-cdn.com/elastic/original/3X/e/8/e87191330ccc8d1068f4a7f3ceff7a22899f44ec.png)

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [December 17, 2019, 12:01pm UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/11 "2019-12-17T12:01:58Z")

</div>

Hi

Could you share a line or two from your actual .csv file?

Could you remove all filters and share the output from `stdout` for those lines?

---

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 17, 2019, 12:31pm UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/12 "2019-12-17T12:31:51Z")

</div>

CSV fields:

"id tweet date author text app id user followers following stauses location urls geolocation name description url\_media type media quoted relation replied\_id user replied retweeted\_id user retweeted quoted\_id user quoted first HT lang created\_at verified avatar link"

CSV lines:

"1202156044983689216 2019-12-04 09:21:50 @BCibada RT @omnium: 🔸 Avui @omniumintl engeguem una campanya a França, al Regne Unit i a Alemanya perquè els ciutadans europeus demanin als seus líders polítics que s'impliquin a trobar una solució política per Catalunya 📲 #ActForCatalonia [https://t.co/Lg1sS7ZRcO](https://t.co/Lg1sS7ZRcO) Twitter for Android 882852012215369732 1173 2482 70875 [https://www.omnium.cat/ca/omnium-interpella-massivament-els-principals-liders-europeus-perque-treballin-a-favor-duna-solucio-politica/](https://www.omnium.cat/ca/omnium-interpella-massivament-els-principals-liders-europeus-perque-treballin-a-favor-duna-solucio-politica/) None Blad Cibada 🎗️ 🏳️‍🌈 💜 Defending Progressive views on Civil Rights, Politics and Economics. None None None RT None None 1202155081589760000 @omnium None None ActForCatalonia ca 2017-07-06 06:41:37 False [https://pbs.twimg.com/profile\_images/971692733818695681/eMFMibIr\_normal.jpg](https://pbs.twimg.com/profile_images/971692733818695681/eMFMibIr_normal.jpg)[https://twitter.com/BCibada/status/1202156044983689216](https://twitter.com/BCibada/status/1202156044983689216)"

"1202155864821567488 2019-12-04 09:21:07 @OmniumIntl 🇩🇪 Wir starten die Kampagne #ActForCatalonia! Wir bitten dabei die Bürger dieser Länder, sich mit ihren Volksvertretern in Verbindung zu setzen, damit diese dabei helfen, eine politische Lösung für Katalonien zu finden: [https://t.co/hYXZMCPqub](https://t.co/hYXZMCPqub) Twitter Web App 1049970228854226946 9350 791 1605 Barcelona, Catalonia [https://www.omnium.cat/de/actforcatalonia/](https://www.omnium.cat/de/actforcatalonia/) None Òmnium International Freedom for the Catalan political prisoners. Catalonia deserves a political solution. None None None reply 1202155863286439936 @OmniumIntl None None None None ActForCatalonia de 2018-10-10 10:29:44 True [https://pbs.twimg.com/profile\_images/1050417217714757633/8R8m4bDO\_normal.jpg](https://pbs.twimg.com/profile_images/1050417217714757633/8R8m4bDO_normal.jpg)[https://twitter.com/OmniumIntl/status/1202155864821567488](https://twitter.com/OmniumIntl/status/1202155864821567488)"

stdout:

 ![imagen](https://us1.discourse-cdn.com/elastic/original/3X/d/8/d85aaed99e4326144a24ceeb245de55e666cbd0d.png)

---

<div class="post-metadata">

**Author:** ![ITIC](https://avatars.discourse-cdn.com/v4/letter/i/90ced4/32.png) [@ITIC](https://discuss.elastic.co/u/ITIC)\
**Post date:** [December 17, 2019, 12:46pm UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/13 "2019-12-17T12:46:36Z")

</div>

Hi

The CSV lines you posted don't contain tabs, only spaces, but that might be due to pasting them here. In your "message", though, the separator is "\t".

Since the CSV filter is parsing your "message", try using "\t" as separator in your CSV filter, or maybe "\\t".

Besides, you have icons (non-character) in your csv (and in your "message"), and that might be giving you a hard time.

Hope this helps.

---

<div class="post-metadata">

**Author:** ![frangolzmil](https://avatars.discourse-cdn.com/v4/letter/f/4af34b/32.png) [@frangolzmil](https://discuss.elastic.co/u/frangolzmil)\
**Post date:** [December 17, 2019, 12:55pm UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/14 "2019-12-17T12:55:27Z")

</div>

Using NotePad.

 ![imagen](https://us1.discourse-cdn.com/elastic/original/3X/b/2/b2e1fd41435b97e7db5c8bd30c8202e43bbeb9a6.png)  
 ![imagen](https://us1.discourse-cdn.com/elastic/original/3X/4/9/4998870d8c4425e3ece589e4a207be18f34eec43.png)

Thank u for your help. I appreciate.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 14, 2020, 12:55pm UTC](https://discuss.elastic.co/t/error-parsing-csv-file-logstash/211963/15 "2020-01-14T12:55:32Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.