Json parsing problem tcp input

Elastic Stack Logstash
1

I have a problem with logs in syslog.
Logstash logging warn like a [WARN ][logstash.codecs.jsonlines] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: No message available>, :data=>"null\r"}
per milisecond.

My configuration :

    input {
  tcp {
    port => 5044
    codec => json
  }
}

filter {
    if "_jsonparsefailure" in [tags] {
	drop { }
	}
    else {
        json {	source => "message"
        	target => "log"
	    }
	}
    }

output {
  elasticsearch {
   hosts => "127.0.0.1:9200"
   manage_template => false
   index => "%{test}-%{+YYYY.MM.dd}"
   }
}

Logs are growing very fast and the elasticsearch service hangs after a long time, which requires a restart.

2

That is telling you that it received a message containing "null\r", which will have resulted in a _jsonparsefailure tag being added and the event being dropped.

3

Does this mean that logstash expects a value other than null?
The data is correctly displayed in the visualizations, however, the occurrence of such an warnings in the logs is irritating and uses a lot of space.
Changing log.level is not what I am expecting at the moment.

4

You have configured the tcp input with a json codec. It expects valid json, which "null\r" is assuredly not.

5

OK, I removed the codec from the input and changed the configuration to the following.

filter {
        if [message] =~ /^\s*$/ {
            drop { }
            }
        json {
            source => "message"
            skip_on_invalid_json => true
            remove_field => ["message"]
            }
        }

However, I still have the same problem. Below is a rubydebug dump.

{
   "@version" => "1",
      "port" => xxxx,
 "@timestamp" => 2019-02-26T10:04:45.537Z,
    "message" => "null\r",
      "host" => "x.x.x.x",
      "tags" => [
    [0] "_jsonparsefailure"
 ]
 }
6

Which is what I would expect. Something is still sending messages containing 'null\r', and a json filter will fail to parse them.

7

Can I do something this without changing the messages? Probably I will get similar messages from other servers which I probably will not be able to influence.

8

Drop them?

if "_jsonparsefailure" in [tags] { drop {} }
9

Ok, I remove json coden in the input and configure it in filter with if "_jsonparsefailure" in [tags] { drop {} }and also managed to change the transmitted frames so that they do not generate so many errors. Problem is solved. Thank you for your advices Badger :slight_smile:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.