Drop empty message with character NULL \x00

[quangnguyen]

Hello Community,

i'm getting following log message what makes no sense to forward it in to output. The message contains only null character inside two qoutes "\x00". How could i drop it with logstash filter?

[2020-03-20T01:08:47,788][ERROR][logstash.inputs.gelf     ][main] JSON parse failure. Falling back to plain-text {:error=>#<L
ogStash::Json::ParserError: Illegal character ((CTRL-CHAR, code 0)): only regular white space (\r, \n, \t) is allowed between tokens at [Source: (byte[])" "; line: 1, column: 2]>, :data=>"\"\\x00\""}
{
        "message" => "\"\\x00\"",
       "@version" => "1",
     "@timestamp" => 2020-03-20T00:08:47.787Z,
    "source_host" => "192.168.99.104",
           "tags" => [
        [0] "_jsonparsefailure"
    ]
}
{
        "message" => "\"\\x00\"",
       "@version" => "1",
     "@timestamp" => 2020-03-20T00:08:47.788Z,
    "source_host" => "192.168.99.104",
           "tags" => [
        [0] "_jsonparsefailure"
    ]
}

I already did following filter but it does not drop the message.

    if [message] =~ /^\"[\s]\"$|^\"\"$|^\"\\x00\"$/ {
       drop { }
    }

[calanon (Chris)]

You could isolate the field then use mutate to filter it out.

  mutate { remove_field => [ "field1", "field2", "field3", ... "fieldN" ] }
}

[Badger]

If the contents of your string were double quotes followed by NUL followed by double quotes then I would expect rubydebug to show it as

   "message" => "\"\u0000\""

That said, I can get the json codec to fail to parse with

#<LogStash::Json::ParserError: Illegal unquoted character ((CTRL-CHAR, code 0)): has to be escaped using backslash to be included in string value
 at [Source: (String)""""; line: 1, column: 3]>, :data=>"\"\u0000\""}

or, if the escape is added,

>#<LogStash::Json::ParserError: Unrecognized character escape (CTRL-CHAR, code 0)
 at [Source: (String)""\""; line: 1, column: 4]>, :data=>"\"\\\u0000\""}

Does it help you to do this?

if "_jsonparsefailure" in [tags] { drop {} }

[quangnguyen]

@Badger It's not the solution for me. In case of _jsonparsefailure, it's ok for me with the fallback to plaintext but i just do want to drop the whole of event if its message contains only "blank" string "\"\\x00\""

The following document should not appear in Elasticsearch at end of the day because it does not have added value for me.

{
  "_index": "logstash-2020.03.19-000001",
  "_type": "_doc",
  "_id": "5QQzBHEBr0m51kUUnDVl",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2020-03-22T21:43:43.332Z",
    "message": "\"\\x00\"",
    "@version": "1",
    "source_host": "192.168.99.104",
    "tags": [
      "_jsonparsefailure"
    ]
  },
  "fields": {
    "@timestamp": [
      "2020-03-22T21:43:43.332Z"
    ]
  },
  "sort": [
    1584913423332
  ]
}