Dealing with invalid json

Elastic Stack Logstash
1

I'm trying to import some json data that is in a column in redshift. Some of the records have a field that includes invalid json escape sequences, such as this:

"work_title": "The Discrete and Semi-continuous Fr\'echet Distance with Shortcuts via Approximate Distance Counting and Selection Techniques"

"work_title": "On the \(\partial\overline{\partial}\)-Lemma and Bott-Chern cohomology"

I can remove the first one with logstash using this:

input {
    jdbc {

    }
}

filter {
      mutate {
        gsub => [
          "json_elastic", "\\'", "'"
        ]
      }
      json {
        source => "json_elastic"
      }
      mutate {
        remove_field => ["json_elastic"]
      }
}

output {
     stdout { }
}

But is there a way I can expand this to remove the other '/'? I tried this but I get a configuration error:

mutate {
        gsub => [
          "json_elastic", "\\", ""
        ]
      }

Or is there a way to not parse json within that field and simply get the text?

2

The logstash configuration compiler will always interpret a backslash before the end of a double quoted string as escaping the double quote.

The standard trick is to use a single occurrence of a character group with a single member, which is equivalent to the single character in the group: "[\\]"

1 Like
3

Awesome, thank you!

4

This should be added in the documentation, i think a lot of people are struggling with the use of \ to match things.

I see you around alot i was wondering if there is something to be done to help troubleshooting logstash common problems ( in the documentation )

5

I do not know how to get the documentation updated. There are a boatload of small issues I would like to see fixed.

6

I think there is a way to update trough GitHub - elastic/logstash-docs: GENERATED REPOSITORY. DO NOT EDIT. - Documentation repository for Logstash static asciidoc and generated plugin asciidoc.

I do not know if the pull requests are accepted tho

7

This is an example of PR that modifies documentation. Like you, I don't know if PRs are accepted from random folk like me.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.