Error parsing json with Logstash

Elastic Stack Logstash
1

Hi there,
I'm using ELK 7.3.1
And getting Error parsing json for "message"
This is how message looks like in origin (Exacmple)-

message	       	{"time_date": "2019-02-14T14:00:39+00:00","client": "10.xxx.xxx.xxx", "host": "xxx.com", "scheme": "https", "request_method": "GET", "request_uri": "/static/img/logo_new.png", "request_id": "xxxxxxxxxxxxxx", "status": 304, "upstream_addr": "xxx.xx.xx.xx:80", "upstream_status": 304, "request_time": 0.002, "upstream_response_time": 0.000, "upstream_connect_time": 0.000, "upstream_header_time": 0.000}

The tag error that I get in Logstash is -

beats_input_codec_plain_applied, _jsonparsefailure

The logs in Logstash -

2019-09-09T17:00:55.642564748Z  at [Source: (byte[])"{"time_date": "2019-09-09T17:00:54+00:00","client": "10.xxx.x.xxx", "host": "pro.pipl.com", "scheme": "https", "request_method": "POST", "request_uri": "/search/", "request_id": "5d54cc2f24c4420fb2dbc49500bcefa9", "status": 499, "upstream_addr": "xxx.xx.xxx.xxx:80", "upstream_status": -, "request_time": 1.679, "upstream_response_time": 1.680, "upstream_connect_time": 0.004, "upstream_header_time": -}"; line: 1, column: 289]>}
2019-09-09T17:02:16.804707218Z [2019-09-09T17:02:16,804][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"{\"time_date\": \"2019-09-09T17:02:15+00:00\",\"client\": \"xx.xxx.x.xxx\", \"host\": \"pipl.com\", \"scheme\": \"https\", \"request_method\": \"GET\", \"request_uri\": \"/locationautocomplete/\", \"request_id\": \"624beb1705dae1c6d6d8b99ac66b7ac3\", \"status\": 499, \"upstream_addr\": \"xxx.xx.xxx.xxx:80\", \"upstream_status\": -, \"request_time\": 0.285, \"upstream_response_time\": 0.284, \"upstream_connect_time\": 0.000, \"upstream_header_time\": -}", :exception=>#<LogStash::Json::ParserError: Unexpected character (',' (code 44)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value

My Logstash conf -

apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-kube-config
data:
  logstash.conf: |-
    input {
        beats {
            port => 5044
        }
    }
    filter {
        if  [kubernetes][container][name] == "nginx-ingress" {

            json {
                source => "message"
                remove_field => "message"
              }

        }

        else if  [kubernetes][container][name] == "nginx" {
           grok {
               match => {
                   "message" => "%{IP:remote_ip} - \[%{HTTPDATE:[response][time]}\] \"%{DATA:url}\" %{NUMBER:[response][code]} %{NUMBER:[response][bytes]} %{QS:user_agent}"
               }
               remove_field => "message"


           }



           geoip {
               source => "remote_ip"
               target => "[geoip]"
           }


       }

       else {
            drop {}
        }

        date {
            match => ["time", "ISO8601"]
            remove_field => ["time"]
        }

        mutate {
            remove_field => ["source", "host", "[beat][name]", "[beat][version]"]
        }
    }

    output {
            elasticsearch {
                hosts => ["http://...:9200"]
                index => "apps-prod-dal10-%{[kubernetes][namespace]}-deployment-%{[kubernetes][container][name]}-%{[kubernetes][replicaset][name]}%{+YYYY.MM.dd}"

        }
    }

What am I doing wrong here? (PS , worked before the upgrade from 6.4...)

Thanks!

Aleksei

2

Please do not post pictures of text. Just post the text of the JSON and the text of the error message.

3

Understood, I edited the original message above

4

That is telling you that

"upstream_status": -,

is not valid JSON. Similarly

"upstream_header_time": -}

You might be able to fix the JSON up using mutate+gsub

5

I see ,
Is it going to be like this in terms of syntax?

filter {
        if  [kubernetes][container][name] == "nginx-ingress" {

            mutate {
                    gsub => [
                      "upstream_status", "[\\?#-]", ".",
                      "upstream_header_time", "[\\?#-}]", "."

                    ]
                  }

            json {
                source => "message"
                remove_field => "message"
              }

        }

        else if  [kubernetes][container][name] == "nginx" {
           grok {
               match => {
                   "message" => "%{IP:remote_ip} - \[%{HTTPDATE:[response][time]}\] \"%{DATA:url}\" %{NUMBER:[response][code]} %{NUMBER:[response][bytes]} %{QS:user_agent}"
               }
               remove_field => "message"


           }
6

After editing it to -

filter {
        if  [kubernetes][container][name] == "nginx-ingress" {

            mutate {
                gsub => ["[message]", "[-]", ""]
                gsub => ["[message]", "[-}]", ""]
            }

            json {
                source => "message"
                remove_field => "message"
              }

        }

I recieve

2019-09-09T18:59:29.191116817Z [2019-09-09T18:59:29,188][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"{\"time_date\": \"20190909T18:59:27+00:00\",\"client\": \"10.xxx.x.xxx\", \"host\": \"pipl.com\", \"scheme\": \"https\", \"request_method\": \"HEAD\", \"request_uri\": \"/health_check\", \"request_id\": \"6164be42477cf89c35cea59e9e8e18f9\", \"status\": 200, \"upstream_addr\": \"xxx.xx.xxx.1:80\", \"upstream_status\": 200, \"request_time\": 0.005, \"upstream_response_time\": 0.004, \"upstream_connect_time\": 0.000, \"upstream_header_time\": 0.004", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"time_date": "20190909T18:59:27+00:00","client": "xx.xxx.3.xxx", "host": "pipl.com", "scheme": "https", "request_method": "HEAD", "request_uri": "/health_check", "request_id": "6164be42477cf89c35cea59e9e8e18f9", "status": 200, "upstream_addr": "xxx.xx.xxx.1:80", "upstream_status": 200, "request_time": 0.005, "upstream_response_time": 0.004, "upstream_connect_time": 0.000, "upstream_header_time": 0.004"; line: 1, column: 1])

What is the correct syntax for it?

7

Try

mutate { gsub => [ "message", "(\W)-(\W)", '\1""\2' ] }
1 Like
8

Tried, now I get

2019-09-09T19:12:44.567181003Z [2019-09-09T19:12:44,562][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"I0909 19:11:58.404865       1 controller.go:276] Endpoints logstash-kube changed, syncing", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'I0909': was expecting 'null', 'true', 'false' or NaN
9

My guess is that something is sending you messages that are not valid JSON. If you want to suppress the error message you can use

skip_on_invalid_json => true

in the json filter.

10

Though it doesn't really resolve the issue, just suppresses the error message...
Well...I will try to dig little bit more

11

By the way, I checked once again the JSON itself, even in JSON validator, and it's totally correct.
Here's a JSON of nginx-ingress (the if statement in logstash file) -

{"time_date": "2019-09-10T03:15:37+00:00","client": "xx.xxx.3.xxx", "host": "api.pipl.com", "scheme": "https", "request_method": "POST", "request_uri": "/apis/gateway/search/", "request_id": "7aee122736f316d38bde693288ce6159", "status": 200, "upstream_addr": "xxx.xx.xxx.xx:80", "upstream_status": 200, "request_time": 0.846, "upstream_response_time": 0.844, "upstream_connect_time": 0.000, "upstream_header_time": 0.844}

It's correct JSON, so I don't know how Logstash decides that it's not.

12

That example is valid JSON and logstash has no problem parsing it.

The problem with

{"time_date": "20190909T18:59:27+00:00","client": "10.xxx.x.xxx", "host": "pipl.com", "scheme": "https", "request_method": "HEAD", "request_uri": "/health_check", "request_id": "6164be42477cf89c35cea59e9e8e18f9", "status": 200, "upstream_addr": "xxx.xx.xxx.1:80", "upstream_status": 200, "request_time": 0.005, "upstream_response_time": 0.004, "upstream_connect_time": 0.000, "upstream_header_time": 0.004

is that it is missing a closing }

13

No, I just didn't copy paste all line, there is a closing }.
The problem is like you mentioned earlier , in "upstream_header_time": -} , the "-" makes it invalid, I'm playing with mutate gsub and mutate replace , to fix the "-" sign with empty space or 0.
Without success till now

14

Hi @Badger!
Actually

mutate { gsub => [ "message", "(\W)-(\W)", '\1""\2' ] }

helped, and JSON is correct now.

The error that I sent you, was problem of logstash parsing the date, though the JSON once again, is correct.

Here is the JSON

{"time_date": "2019-09-11T07:05:32+00:00", "client": "10.176.3.115", "host": "pipl.com", "scheme": "https", "request_method": "GET", "request_uri": "/rd/", "request_id": "f44d35bcf367be11c2804fd2f7d2dd41", "status": 499, "upstream_addr": "111.11.111.111:80", "-": -1, "request_time": 0.295, "upstream_response_time": 0.296, "upstream_connect_time": 0.004, "upstream_header_time": -1}

And the Logstash Error is -

controller.go:276] Endpoints logstash-kube changed, syncing"; line: 1, column: 7]>}
2019-09-11T09:49:06.908134227Z [2019-09-11T09:49:06,907][WARN ][logstash.filters.json    ] Error parsing json {:source=>"message", :raw=>"I0911 09:48:20.464574       1 controller.go:276]

That means, that it can't parse the date - raw=>"I0911.
Any idea why?

Can it be resolved with Date filter plugin for Logstash?

15

@Badger

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.