Error parsing json with Logstash

Hi there,
I'm using ELK 7.3.1
And getting Error parsing json for "message"
This is how message looks like in origin (Exacmple)-

message	       	{"time_date": "2019-02-14T14:00:39+00:00","client": "10.xxx.xxx.xxx", "host": "xxx.com", "scheme": "https", "request_method": "GET", "request_uri": "/static/img/logo_new.png", "request_id": "xxxxxxxxxxxxxx", "status": 304, "upstream_addr": "xxx.xx.xx.xx:80", "upstream_status": 304, "request_time": 0.002, "upstream_response_time": 0.000, "upstream_connect_time": 0.000, "upstream_header_time": 0.000}

The tag error that I get in Logstash is -

beats_input_codec_plain_applied, _jsonparsefailure

The logs in Logstash -

2019-09-09T17:00:55.642564748Z  at [Source: (byte[])"{"time_date": "2019-09-09T17:00:54+00:00","client": "10.xxx.x.xxx", "host": "pro.pipl.com", "scheme": "https", "request_method": "POST", "request_uri": "/search/", "request_id": "5d54cc2f24c4420fb2dbc49500bcefa9", "status": 499, "upstream_addr": "xxx.xx.xxx.xxx:80", "upstream_status": -, "request_time": 1.679, "upstream_response_time": 1.680, "upstream_connect_time": 0.004, "upstream_header_time": -}"; line: 1, column: 289]>}
2019-09-09T17:02:16.804707218Z [2019-09-09T17:02:16,804][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"{\"time_date\": \"2019-09-09T17:02:15+00:00\",\"client\": \"xx.xxx.x.xxx\", \"host\": \"pipl.com\", \"scheme\": \"https\", \"request_method\": \"GET\", \"request_uri\": \"/locationautocomplete/\", \"request_id\": \"624beb1705dae1c6d6d8b99ac66b7ac3\", \"status\": 499, \"upstream_addr\": \"xxx.xx.xxx.xxx:80\", \"upstream_status\": -, \"request_time\": 0.285, \"upstream_response_time\": 0.284, \"upstream_connect_time\": 0.000, \"upstream_header_time\": -}", :exception=>#<LogStash::Json::ParserError: Unexpected character (',' (code 44)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value

My Logstash conf -

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-kube-config
data:
  logstash.conf: |-
    input {
        beats {
            port => 5044
        }
    }
    filter {
        if  [kubernetes][container][name] == "nginx-ingress" {

            json {
                source => "message"
                remove_field => "message"
              }

        }

        else if  [kubernetes][container][name] == "nginx" {
           grok {
               match => {
                   "message" => "%{IP:remote_ip} - \[%{HTTPDATE:[response][time]}\] \"%{DATA:url}\" %{NUMBER:[response][code]} %{NUMBER:[response][bytes]} %{QS:user_agent}"
               }
               remove_field => "message"


           }



           geoip {
               source => "remote_ip"
               target => "[geoip]"
           }


       }

       else {
            drop {}
        }

        date {
            match => ["time", "ISO8601"]
            remove_field => ["time"]
        }

        mutate {
            remove_field => ["source", "host", "[beat][name]", "[beat][version]"]
        }
    }

    output {
            elasticsearch {
                hosts => ["http://...:9200"]
                index => "apps-prod-dal10-%{[kubernetes][namespace]}-deployment-%{[kubernetes][container][name]}-%{[kubernetes][replicaset][name]}%{+YYYY.MM.dd}"

        }
    }

What am I doing wrong here? (PS , worked before the upgrade from 6.4...)

Thanks!

Aleksei

Please do not post pictures of text. Just post the text of the JSON and the text of the error message.

Understood, I edited the original message above

That is telling you that

"upstream_status": -,

is not valid JSON. Similarly

"upstream_header_time": -}

You might be able to fix the JSON up using mutate+gsub

I see ,
Is it going to be like this in terms of syntax?

filter {
        if  [kubernetes][container][name] == "nginx-ingress" {

            mutate {
                    gsub => [
                      "upstream_status", "[\\?#-]", ".",
                      "upstream_header_time", "[\\?#-}]", "."

                    ]
                  }

            json {
                source => "message"
                remove_field => "message"
              }

        }

        else if  [kubernetes][container][name] == "nginx" {
           grok {
               match => {
                   "message" => "%{IP:remote_ip} - \[%{HTTPDATE:[response][time]}\] \"%{DATA:url}\" %{NUMBER:[response][code]} %{NUMBER:[response][bytes]} %{QS:user_agent}"
               }
               remove_field => "message"


           }

After editing it to -

filter {
        if  [kubernetes][container][name] == "nginx-ingress" {

            mutate {
                gsub => ["[message]", "[-]", ""]
                gsub => ["[message]", "[-}]", ""]
            }

            json {
                source => "message"
                remove_field => "message"
              }

        }

I recieve

2019-09-09T18:59:29.191116817Z [2019-09-09T18:59:29,188][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"{\"time_date\": \"20190909T18:59:27+00:00\",\"client\": \"10.xxx.x.xxx\", \"host\": \"pipl.com\", \"scheme\": \"https\", \"request_method\": \"HEAD\", \"request_uri\": \"/health_check\", \"request_id\": \"6164be42477cf89c35cea59e9e8e18f9\", \"status\": 200, \"upstream_addr\": \"xxx.xx.xxx.1:80\", \"upstream_status\": 200, \"request_time\": 0.005, \"upstream_response_time\": 0.004, \"upstream_connect_time\": 0.000, \"upstream_header_time\": 0.004", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"time_date": "20190909T18:59:27+00:00","client": "xx.xxx.3.xxx", "host": "pipl.com", "scheme": "https", "request_method": "HEAD", "request_uri": "/health_check", "request_id": "6164be42477cf89c35cea59e9e8e18f9", "status": 200, "upstream_addr": "xxx.xx.xxx.1:80", "upstream_status": 200, "request_time": 0.005, "upstream_response_time": 0.004, "upstream_connect_time": 0.000, "upstream_header_time": 0.004"; line: 1, column: 1])

What is the correct syntax for it?

Try

mutate { gsub => [ "message", "(\W)-(\W)", '\1""\2' ] }

Tried, now I get

2019-09-09T19:12:44.567181003Z [2019-09-09T19:12:44,562][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"I0909 19:11:58.404865       1 controller.go:276] Endpoints logstash-kube changed, syncing", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'I0909': was expecting 'null', 'true', 'false' or NaN

My guess is that something is sending you messages that are not valid JSON. If you want to suppress the error message you can use

skip_on_invalid_json => true

in the json filter.

Though it doesn't really resolve the issue, just suppresses the error message...
Well...I will try to dig little bit more

By the way, I checked once again the JSON itself, even in JSON validator, and it's totally correct.
Here's a JSON of nginx-ingress (the if statement in logstash file) -

{"time_date": "2019-09-10T03:15:37+00:00","client": "xx.xxx.3.xxx", "host": "api.pipl.com", "scheme": "https", "request_method": "POST", "request_uri": "/apis/gateway/search/", "request_id": "7aee122736f316d38bde693288ce6159", "status": 200, "upstream_addr": "xxx.xx.xxx.xx:80", "upstream_status": 200, "request_time": 0.846, "upstream_response_time": 0.844, "upstream_connect_time": 0.000, "upstream_header_time": 0.844}

It's correct JSON, so I don't know how Logstash decides that it's not.

That example is valid JSON and logstash has no problem parsing it.

The problem with

{"time_date": "20190909T18:59:27+00:00","client": "10.xxx.x.xxx", "host": "pipl.com", "scheme": "https", "request_method": "HEAD", "request_uri": "/health_check", "request_id": "6164be42477cf89c35cea59e9e8e18f9", "status": 200, "upstream_addr": "xxx.xx.xxx.1:80", "upstream_status": 200, "request_time": 0.005, "upstream_response_time": 0.004, "upstream_connect_time": 0.000, "upstream_header_time": 0.004

is that it is missing a closing }

No, I just didn't copy paste all line, there is a closing }.
The problem is like you mentioned earlier , in "upstream_header_time": -} , the "-" makes it invalid, I'm playing with mutate gsub and mutate replace , to fix the "-" sign with empty space or 0.
Without success till now

Hi @Badger!
Actually

mutate { gsub => [ "message", "(\W)-(\W)", '\1""\2' ] }

helped, and JSON is correct now.

The error that I sent you, was problem of logstash parsing the date, though the JSON once again, is correct.

Here is the JSON

{"time_date": "2019-09-11T07:05:32+00:00", "client": "10.176.3.115", "host": "pipl.com", "scheme": "https", "request_method": "GET", "request_uri": "/rd/", "request_id": "f44d35bcf367be11c2804fd2f7d2dd41", "status": 499, "upstream_addr": "111.11.111.111:80", "-": -1, "request_time": 0.295, "upstream_response_time": 0.296, "upstream_connect_time": 0.004, "upstream_header_time": -1}

And the Logstash Error is -

controller.go:276] Endpoints logstash-kube changed, syncing"; line: 1, column: 7]>}
2019-09-11T09:49:06.908134227Z [2019-09-11T09:49:06,907][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"I0911 09:48:20.464574       1 controller.go:276]

That means, that it can't parse the date - raw=>"I0911.
Any idea why?

Can it be resolved with Date filter plugin for Logstash?

@Badger

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.