# Logstash: MalformedCSVError

**URL:** <https://discuss.elastic.co/t/logstash-malformedcsverror/340330>\
**Category:** Logstash\
**Created:** [August 8, 2023, 7:08am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330 "2023-08-08T07:08:27Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![benhartwich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benhartwich/32/124418_2.png) [@benhartwich](https://discuss.elastic.co/u/benhartwich)\
**Post date:** [August 8, 2023, 7:08am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/1 "2023-08-08T07:08:27Z")

</div>

Hi,

can anybody help me to find the right mutate =\> gsub definition to avoid these warnings / errors:

```auto
[WARN] 2023-08-08 07:05:15.003 [[main]>worker20] csv - Error parsing csv {:field=>"message", :source=>"16600,26200,\" 2004 hat Snow Patrol \\\"Run\\\" rausgebracht und nur vier Jahre später hat diesen Song\"", :exception=>#<CSV::MalformedCSVError: Any value after quoted field isn't allowed in line 1.>}

```

I´ve tried:

```auto
  mutate {
    gsub => ["message","\"","'"]

```

but it doesn´t work. Thanks!

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [August 8, 2023, 8:19am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/2 "2023-08-08T08:19:34Z")

</div>

Welcome to the community!

I have assumed this is your message:

> [@benhartwich](#):
>
> `"16600,26200,\" 2004 hat Snow Patrol \\\"Run\\\" rausgebracht und nur vier Jahre später hat diesen Song\""`

You can use this:

```auto
 	mutate{ gsub => ["message", '\"', "'"] }
 	mutate{ gsub => ["message", "[\\]", '' ] }

```

To get:  
`"message" => "16600,26200,' 2004 hat Snow Patrol 'Run' rausgebracht und nur vier Jahre später hat diesen Song'"`

---

<div class="post-metadata">

**Author:** ![benhartwich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benhartwich/32/124418_2.png) [@benhartwich](https://discuss.elastic.co/u/benhartwich)\
**Post date:** [August 8, 2023, 8:20am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/3 "2023-08-08T08:20:13Z")

</div>

Thanks, so I need both?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [August 8, 2023, 8:22am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/4 "2023-08-08T08:22:12Z")

</div>

Yes, because without 2nd gsub, you will get:  
`message" => "16600,26200,\\' 2004 hat Snow Patrol \\\\\\'Run\\\\\\' rausgebracht und nur vier Jahre später hat diesen Song\\'"`

---

<div class="post-metadata">

**Author:** ![benhartwich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benhartwich/32/124418_2.png) [@benhartwich](https://discuss.elastic.co/u/benhartwich)\
**Post date:** [August 8, 2023, 8:28am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/5 "2023-08-08T08:28:10Z")

</div>

Thanks, so this is my config:

```auto
input {
  file {
    path => "/home/ai-upload/vtt/*.csv" 
    start_position => "beginning"
    sincedb_path => "/home/ai-upload/sincedb" 
  }
}

filter {
  csv {
    separator => ","
    columns => ["start", "end", "text", "Timestamp"]
  }
  mutate{ gsub => ["message", '\"', "'"] }
  mutate{ gsub => ["message", "[\\]", '' ] }
  mutate { remove_field => ["path", "host", "@timestamp", "@version"] }
}

output {
  elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "moderationen"
  }
}

```

I´ve tested it, but the errors persist:

```auto
[WARN] 2023-08-08 08:23:19.087 [[main]>worker31] csv - Error parsing csv {:field=>"message", :source=>"15000,19000,\" The Boss Hoss kennt ihr schon von ihrem letzten Megahit, nämlich \\\"Little Help\\\".\"", :exception=>#<CSV::MalformedCSVError: Any value after quoted field isn't allowed in line 1.>}

```

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [August 8, 2023, 8:38am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/6 "2023-08-08T08:38:04Z")

</div>

Try like this, first replace:

```auto
filter {

  mutate{ gsub => ["message", '\"', "'"] }
  mutate{ gsub => ["message", "[\\]", '' ] }

  csv {
    separator => ","
    columns => ["start", "end", "text", "Timestamp"]
  }

```

---

<div class="post-metadata">

**Author:** ![benhartwich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benhartwich/32/124418_2.png) [@benhartwich](https://discuss.elastic.co/u/benhartwich)\
**Post date:** [August 8, 2023, 8:45am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/7 "2023-08-08T08:45:25Z")

</div>

Thanks, this works, but now the columns are splitted different. It seems that the mutation generates a new line seperator. Can I avoid this?

---

<div class="post-metadata">

**Author:** ![Rios](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rios/32/95745_2.png) [@Rios](https://discuss.elastic.co/u/Rios)\
**Post date:** [August 8, 2023, 8:51am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/8 "2023-08-08T08:51:36Z")

</div>

You have 3 columns:  
` columns => ["start", "end", "text"]`

I don't see the _Timestamp_ column in samples.

---

<div class="post-metadata">

**Author:** ![benhartwich](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benhartwich/32/124418_2.png) [@benhartwich](https://discuss.elastic.co/u/benhartwich)\
**Post date:** [August 8, 2023, 9:03am UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/9 "2023-08-08T09:03:58Z")

</div>

I´ve checked that. Now the script includes the column Timestamp. I think it is the semicolon within the column text - e.g.:

```auto
53760,59640," Wenn ihr wisst, wo es staut oder wo vielleicht geblitzt wird, ruft an 07 32 3 mal die 7 3",Wed 10 May 2023 04:56:33 AM CEST

```

Is there another mutation necessary?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [August 8, 2023, 8:20pm UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/10 "2023-08-08T20:20:56Z")

</div>

> [@benhartwich](#):
>
> `53760,59640," Wenn ihr wisst, wo es staut oder wo vielleicht geblitzt wird, ruft an 07 32 3 mal die 7 3",Wed 10 May 2023 04:56:33 AM CEST`

In that case you would use

```
    mutate{ gsub => ["message", "'", '"'] }
    csv {}

```

If you also have lines like the first one you shared you may need to make the mutates conditional.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 5, 2023, 8:21pm UTC](https://discuss.elastic.co/t/logstash-malformedcsverror/340330/11 "2023-09-05T20:21:55Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.