# Error parsing json with Logstash

**URL:** https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749
**Category:** Logstash
**Created:** [September 9, 2019, 4:56pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749 "2019-09-09T16:56:56Z")
**Posts on this page:** 16
**Page:** 1

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 9, 2019, 4:56pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/1 "2019-09-09T16:56:56Z")

</div>

Hi there,  
I'm using ELK 7.3.1  
And getting Error parsing json for "message"  
This is how message looks like in origin (Exacmple)-

```auto
message {"time_date": "2019-02-14T14:00:39+00:00","client": "10.xxx.xxx.xxx", "host": "xxx.com", "scheme": "https", "request_method": "GET", "request_uri": "/static/img/logo_new.png", "request_id": "xxxxxxxxxxxxxx", "status": 304, "upstream_addr": "xxx.xx.xx.xx:80", "upstream_status": 304, "request_time": 0.002, "upstream_response_time": 0.000, "upstream_connect_time": 0.000, "upstream_header_time": 0.000}

```

The tag error that I get in Logstash is -

```auto
beats_input_codec_plain_applied, _jsonparsefailure

```

The logs in Logstash -

```auto
2019-09-09T17:00:55.642564748Z at [Source: (byte[])"{"time_date": "2019-09-09T17:00:54+00:00","client": "10.xxx.x.xxx", "host": "pro.pipl.com", "scheme": "https", "request_method": "POST", "request_uri": "/search/", "request_id": "5d54cc2f24c4420fb2dbc49500bcefa9", "status": 499, "upstream_addr": "xxx.xx.xxx.xxx:80", "upstream_status": -, "request_time": 1.679, "upstream_response_time": 1.680, "upstream_connect_time": 0.004, "upstream_header_time": -}"; line: 1, column: 289]>}
2019-09-09T17:02:16.804707218Z [2019-09-09T17:02:16,804][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"{\"time_date\": \"2019-09-09T17:02:15+00:00\",\"client\": \"xx.xxx.x.xxx\", \"host\": \"pipl.com\", \"scheme\": \"https\", \"request_method\": \"GET\", \"request_uri\": \"/locationautocomplete/\", \"request_id\": \"624beb1705dae1c6d6d8b99ac66b7ac3\", \"status\": 499, \"upstream_addr\": \"xxx.xx.xxx.xxx:80\", \"upstream_status\": -, \"request_time\": 0.285, \"upstream_response_time\": 0.284, \"upstream_connect_time\": 0.000, \"upstream_header_time\": -}", :exception=>#<LogStash::Json::ParserError: Unexpected character (',' (code 44)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value

```

My Logstash conf -

```auto
apiVersion: v1
kind: ConfigMap
metadata:
  name: logstash-kube-config
data:
  logstash.conf: |-
    input {
        beats {
            port => 5044
        }
    }
    filter {
        if [kubernetes][container][name] == "nginx-ingress" {

            json {
                source => "message"
                remove_field => "message"
              }

        }

        else if [kubernetes][container][name] == "nginx" {
           grok {
               match => {
                   "message" => "%{IP:remote_ip} - \[%{HTTPDATE:[response][time]}\] \"%{DATA:url}\" %{NUMBER:[response][code]} %{NUMBER:[response][bytes]} %{QS:user_agent}"
               }
               remove_field => "message"

           }

           geoip {
               source => "remote_ip"
               target => "[geoip]"
           }

       }

       else {
            drop {}
        }

        date {
            match => ["time", "ISO8601"]
            remove_field => ["time"]
        }

        mutate {
            remove_field => ["source", "host", "[beat][name]", "[beat][version]"]
        }
    }

    output {
            elasticsearch {
                hosts => ["http://...:9200"]
                index => "apps-prod-dal10-%{[kubernetes][namespace]}-deployment-%{[kubernetes][container][name]}-%{[kubernetes][replicaset][name]}%{+YYYY.MM.dd}"

        }
    }

```

What am I doing wrong here? (PS , worked before the upgrade from 6.4...)

Thanks!

Aleksei

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 9, 2019, 5:02pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/2 "2019-09-09T17:02:08Z")

</div>

Please do not post pictures of text. Just post the text of the JSON and the text of the error message.

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 9, 2019, 5:32pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/3 "2019-09-09T17:32:46Z")

</div>

Understood, I edited the original message above

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 9, 2019, 5:56pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/4 "2019-09-09T17:56:33Z")

</div>

> [@aleksei.saiko](#):
>
> Unexpected character (',' (code 44)) in numeric value: expected digit (0-9) to follow minus sign, for valid numeric value

That is telling you that

```
"upstream_status": -,

```

is not valid JSON. Similarly

```
"upstream_header_time": -}

```

You might be able to fix the JSON up using mutate+gsub

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 9, 2019, 6:20pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/5 "2019-09-09T18:20:50Z")

</div>

> [@Badger](#):
>
> upstream\_status

I see ,  
Is it going to be like this in terms of syntax?

```auto
filter {
        if [kubernetes][container][name] == "nginx-ingress" {

            mutate {
                    gsub => [
                      "upstream_status", "[\\?#-]", ".",
                      "upstream_header_time", "[\\?#-}]", "."

                    ]
                  }

            json {
                source => "message"
                remove_field => "message"
              }

        }

        else if [kubernetes][container][name] == "nginx" {
           grok {
               match => {
                   "message" => "%{IP:remote_ip} - \[%{HTTPDATE:[response][time]}\] \"%{DATA:url}\" %{NUMBER:[response][code]} %{NUMBER:[response][bytes]} %{QS:user_agent}"
               }
               remove_field => "message"

           }

```

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 9, 2019, 7:01pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/6 "2019-09-09T19:01:29Z")

</div>

After editing it to -

```auto
filter {
        if [kubernetes][container][name] == "nginx-ingress" {

            mutate {
                gsub => ["[message]", "[-]", ""]
                gsub => ["[message]", "[-}]", ""]
            }

            json {
                source => "message"
                remove_field => "message"
              }

        }

```

I recieve

```auto
2019-09-09T18:59:29.191116817Z [2019-09-09T18:59:29,188][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"{\"time_date\": \"20190909T18:59:27+00:00\",\"client\": \"10.xxx.x.xxx\", \"host\": \"pipl.com\", \"scheme\": \"https\", \"request_method\": \"HEAD\", \"request_uri\": \"/health_check\", \"request_id\": \"6164be42477cf89c35cea59e9e8e18f9\", \"status\": 200, \"upstream_addr\": \"xxx.xx.xxx.1:80\", \"upstream_status\": 200, \"request_time\": 0.005, \"upstream_response_time\": 0.004, \"upstream_connect_time\": 0.000, \"upstream_header_time\": 0.004", :exception=>#<LogStash::Json::ParserError: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (byte[])"{"time_date": "20190909T18:59:27+00:00","client": "xx.xxx.3.xxx", "host": "pipl.com", "scheme": "https", "request_method": "HEAD", "request_uri": "/health_check", "request_id": "6164be42477cf89c35cea59e9e8e18f9", "status": 200, "upstream_addr": "xxx.xx.xxx.1:80", "upstream_status": 200, "request_time": 0.005, "upstream_response_time": 0.004, "upstream_connect_time": 0.000, "upstream_header_time": 0.004"; line: 1, column: 1])

```

What is the correct syntax for it?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 9, 2019, 7:05pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/7 "2019-09-09T19:05:56Z")

</div>

Try

```
mutate { gsub => ["message", "(\W)-(\W)", '\1""\2'] }
```

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 9, 2019, 7:14pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/8 "2019-09-09T19:14:28Z")

</div>

> [@Badger](#):
>
> mutate { gsub =\> ["message", "(\W)-(\W)", '\1""\2'] }

Tried, now I get

```auto
2019-09-09T19:12:44.567181003Z [2019-09-09T19:12:44,562][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"I0909 19:11:58.404865 1 controller.go:276] Endpoints logstash-kube changed, syncing", :exception=>#<LogStash::Json::ParserError: Unrecognized token 'I0909': was expecting 'null', 'true', 'false' or NaN

```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 9, 2019, 7:46pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/9 "2019-09-09T19:46:48Z")

</div>

My guess is that something is sending you messages that are not valid JSON. If you want to suppress the error message you can use

```
skip_on_invalid_json => true

```

in the json filter.

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 9, 2019, 8:19pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/10 "2019-09-09T20:19:53Z")

</div>

Though it doesn't really resolve the issue, just suppresses the error message...  
Well...I will try to dig little bit more

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 10, 2019, 5:23am UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/11 "2019-09-10T05:23:44Z")

</div>

By the way, I checked once again the JSON itself, even in JSON validator, and it's totally correct.  
Here's a JSON of nginx-ingress (the if statement in logstash file) -

```auto
{"time_date": "2019-09-10T03:15:37+00:00","client": "xx.xxx.3.xxx", "host": "api.pipl.com", "scheme": "https", "request_method": "POST", "request_uri": "/apis/gateway/search/", "request_id": "7aee122736f316d38bde693288ce6159", "status": 200, "upstream_addr": "xxx.xx.xxx.xx:80", "upstream_status": 200, "request_time": 0.846, "upstream_response_time": 0.844, "upstream_connect_time": 0.000, "upstream_header_time": 0.844}

```

It's correct JSON, so I don't know how Logstash decides that it's not.

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [September 10, 2019, 11:56am UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/12 "2019-09-10T11:56:12Z")

</div>

> [@aleksei.saiko](#):
>
> It's correct JSON

That example is valid JSON and logstash has no problem parsing it.

The problem with

```
{"time_date": "20190909T18:59:27+00:00","client": "10.xxx.x.xxx", "host": "pipl.com", "scheme": "https", "request_method": "HEAD", "request_uri": "/health_check", "request_id": "6164be42477cf89c35cea59e9e8e18f9", "status": 200, "upstream_addr": "xxx.xx.xxx.1:80", "upstream_status": 200, "request_time": 0.005, "upstream_response_time": 0.004, "upstream_connect_time": 0.000, "upstream_header_time": 0.004

```

is that it is missing a closing }

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 10, 2019, 2:49pm UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/13 "2019-09-10T14:49:43Z")

</div>

No, I just didn't copy paste all line, there is a closing }.  
The problem is like you mentioned earlier , in "upstream\_header\_time": -} , the "-" makes it invalid, I'm playing with mutate gsub and mutate replace , to fix the "-" sign with empty space or 0.  
Without success till now

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 11, 2019, 9:54am UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/14 "2019-09-11T09:54:00Z")

</div>

Hi @Badger!  
Actually

```auto
mutate { gsub => ["message", "(\W)-(\W)", '\1""\2'] }

```

helped, and JSON is correct now.

The error that I sent you, was problem of logstash parsing the date, though the JSON once again, is correct.

Here is the JSON

```auto
{"time_date": "2019-09-11T07:05:32+00:00", "client": "10.176.3.115", "host": "pipl.com", "scheme": "https", "request_method": "GET", "request_uri": "/rd/", "request_id": "f44d35bcf367be11c2804fd2f7d2dd41", "status": 499, "upstream_addr": "111.11.111.111:80", "-": -1, "request_time": 0.295, "upstream_response_time": 0.296, "upstream_connect_time": 0.004, "upstream_header_time": -1}

```

And the Logstash Error is -

```auto
controller.go:276] Endpoints logstash-kube changed, syncing"; line: 1, column: 7]>}
2019-09-11T09:49:06.908134227Z [2019-09-11T09:49:06,907][WARN][logstash.filters.json] Error parsing json {:source=>"message", :raw=>"I0911 09:48:20.464574 1 controller.go:276]

```

That means, that it can't parse the date - raw=\>"I0911.  
Any idea why?

Can it be resolved with Date filter plugin for Logstash?

---

<div class="post-metadata">

### Author: ![aleksei.saiko](https://avatars.discourse-cdn.com/v4/letter/a/b9bd4f/32.png) [@aleksei.saiko](https://discuss.elastic.co/u/aleksei.saiko)
#### Post date: [September 12, 2019, 11:32am UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/15 "2019-09-12T11:32:10Z")

</div>

@Badger

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 10, 2019, 11:32am UTC](https://discuss.elastic.co/t/error-parsing-json-with-logstash/198749/16 "2019-10-10T11:32:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.