Proper way to escape escape characters

Elastic Stack Logstash
1

Hello, I have a weird problem. I'm trying to replace \x5c with \ in a mutate. However, the logstash config fails to parse when I do this:

input {
    beats {
      # The port to listen on for filebeat connections.
      port => 5044
      # The IP address to listen for filebeat connections.
      host => "localhost"
    }
}
filter {
    mutate { gsub => [ "message", "\\x5C", '\\' ] }
    json { source => "message" }
}
output {
    stdout { codec => rubydebug }
 }

Data looks like this:
{"test_json": "test\x5C"json\x5C"data"}

I encounter the following error:

[ERROR] 2023-01-27 17:00:35.027 [Converge PipelineAction::Create<main>] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of \"\\\\'\", any character, \"'\" at line 20, column 1 (byte 323) after filter {\n\n    mutate { gsub => [ \"message\", \"\\\\x5C\", '\\\\' ] }\n\n\n    json { source => \"message\" }\n}\noutput {\n\n    stdout { codec => rubydebug }\n }\n", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in `compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:189:in `initialize'", "org/logstash/execution/JavaBasePipelineExt.java:72:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:48:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:392:in `block in converge_state'"]}

It seems that its escaping the ' character causing the config to fail to parse rather than being interpreted like it should as a single \ escape character. Is there a different way I need to escape escape characters? I find it strange I cant use a standard \\ as an escape sequence. I've tried using odd number of escape chars like \ or \\\ and also with \\\\ but it also fails to parse this configuration in either case. It seems no matter what I do it escapes the ' or " in the logstash configuration file causing it to fail to parse the configuration.

Strangely enough I can just remove the binary \x5c with:

mutate { gsub => [ "message", "\\x5C", "" ] }

And that parses just fine. Really scratching my head on this one.

2

The way the logstash configuration parser works you cannot have a backslash at the end of a quoted string. It escapes the closing quote causing the parser to keep consuming your configuration as if it were part of the string, right up until the next unescaped quote or end of file.

See here for a solution.

3

Why are you replacing \x5c with \? Actually it is replacement 'x5c' with nothing :slight_smile:
Additional problem is an improper JSON formatting.
{"test_json": "test\x5C"json\x5C"data"} - quotes inside quotes. Depend on source if is wrong, you can replace double " with single quotes. 2nd gsub replacement is optional.
Try this:

input {
  generator {
       "message" => "test\x5C\"json\x5C\"data"
       count => 1
  }
}
filter {

mutate { gsub => [ "message", "x5C", "" ,
"message", '[\"]', "'",
"message","[\\]",""
]  }

}
output {
    stdout { codec => rubydebug{ }   }
}

Result

{
       "message" => "test\"json\"data",
         "event" => {
        "original" => "test\\x5C\\\"json\\x5C\\\"data"
    }
}

Result with '

{
       "message" => "test'json'data",
         "event" => {
        "original" => "test\\x5C\\\"json\\x5C\\\"data"
    }
}
4

Thank you, this is the solution I needed.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.