# Logstash - elasticsarch and double quotes

**URL:** <https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133>\
**Category:** Logstash\
**Created:** [April 14, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133 "2021-04-14T16:54:59Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![cibernicola](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cibernicola/32/85621_2.png) [@cibernicola](https://discuss.elastic.co/u/cibernicola)\
**Post date:** [April 14, 2021, 4:54pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/1 "2021-04-14T16:54:59Z")

</div>

Hello  
I'm trying to ingest logs wich have " in some of their fields as part of a string, I'm trying to scape them with \ or \ or \\ but no luck, so, is there any way to ingest double quotes from logs into elasticsearch using logstash?

Thanks.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 14, 2021, 6:44pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/2 "2021-04-14T18:44:01Z")

</div>

Why do you feel the need to escape them?

---

<div class="post-metadata">

**Author:** ![cibernicola](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cibernicola/32/85621_2.png) [@cibernicola](https://discuss.elastic.co/u/cibernicola)\
**Post date:** [April 14, 2021, 6:46pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/3 "2021-04-14T18:46:56Z")

</div>

Because I'm seeing errors in output when logstash is trying to "face" a field like: charCHar"charCHAr

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 14, 2021, 7:01pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/4 "2021-04-14T19:01:51Z")

</div>

What errors are you seeing?

---

<div class="post-metadata">

**Author:** ![cibernicola](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cibernicola/32/85621_2.png) [@cibernicola](https://discuss.elastic.co/u/cibernicola)\
**Post date:** [April 14, 2021, 7:06pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/5 "2021-04-14T19:06:52Z")

</div>

For example:

> field=\>"message", :source=\>"string,string"19\r", :exception=\>#\<CSV::MalformedCSVError: Illegal quoting in line 1.\>}

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 14, 2021, 7:15pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/6 "2021-04-14T19:15:59Z")

</div>

The CSV format requires that if a field is quoted (presumably because it may contain a comma) then the entire field must be quoted. That is, the first and last characters must be quotes. You cannot have additional characters such as a carriage return at the end of the field.

If the problem is just a \r at the end of the line you can remove it using mutate+gsub.

---

<div class="post-metadata">

**Author:** ![cibernicola](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cibernicola/32/85621_2.png) [@cibernicola](https://discuss.elastic.co/u/cibernicola)\
**Post date:** [April 14, 2021, 7:22pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/7 "2021-04-14T19:22:36Z")

</div>

> [@Badger](#):
>
> The CSV format requires that if a field is quoted (presumably because it may contain a comma) then the entire field must be quoted. That is, the first and last characters must be quotes. You cannot have additional characters such as a carriage return at the end of the field.
> 
> If the problem is just a \r at the end of the line you can remove it using mutate+gs

I'll try, but the problem is, I guess, that the fuield is not quoted in the csv file, it may contain X double quote(s) or 0 in a field, like this:

```
fieldstringA, field"str""ingB\r
fieldstringA, fieldstringB\r
fieldstringA, fieldst"ringB\r

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 14, 2021, 8:44pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/8 "2021-04-14T20:44:38Z")

</div>

If the fields never contain commas then another option is to replace the quotes with something else, run the csv filter, then gsub them back. I have not tested it, but something like

```
mutate { gsub => ["message", '"', ":#$%^^&"] }
csv { ... }
ruby {
    code => '
        event.to_hash.each { |k, v|
            if v.is_a? String
                event.remove(k)
                event.set(k, v.gsub(":#$%^^&", "\""))
            end
        }
    '
}
```

---

<div class="post-metadata">

**Author:** ![cibernicola](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cibernicola/32/85621_2.png) [@cibernicola](https://discuss.elastic.co/u/cibernicola)\
**Post date:** [April 15, 2021, 7:58am UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/9 "2021-04-15T07:58:06Z")

</div>

So, is not possible to ingest doublequotes? That's my problem, double quotes are part of some fields and what I need is to ingest them ☹

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 15, 2021, 3:07pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/10 "2021-04-15T15:07:53Z")

</div>

> [@cibernicola](#):
>
> So, is not possible to ingest doublequotes?

It is certainly possible to ingest double quotes, but if you want to use a csv filter then the message has to be a properly formatted CSV line. And "properly formatted" [constrains](https://tools.ietf.org/html/rfc4180) the ways in which double quotes can appear.

---

<div class="post-metadata">

**Author:** ![cibernicola](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cibernicola/32/85621_2.png) [@cibernicola](https://discuss.elastic.co/u/cibernicola)\
**Post date:** [April 15, 2021, 4:30pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/11 "2021-04-15T16:30:24Z")

</div>

In fact the file I am trying to import is simply a list of text strings, formed by two values and separated by a constant value, without headers or too much complexity. Would it be perhaps interesting to use another import format, such as Json?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 15, 2021, 4:49pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/12 "2021-04-15T16:49:21Z")

</div>

> [@cibernicola](#):
>
> two values and separated by a constant value

You might be better off replacing the csv filter with dissect if I understood that correctly.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 13, 2021, 4:49pm UTC](https://discuss.elastic.co/t/logstash-elasticsarch-and-double-quotes/270133/13 "2021-05-13T16:49:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.