One of my nginx access log report "_jsonparsefailure" after add to elk?

Elastic Stack Logstash
1

Here is one piece of my nginx access log:

{"@timestamp":"2017-07-22T17:14:23+08:00","host":"117.119.33.237","clientip":"182.246.61.241","remote_user":"-","request":"GET /s?slot=-1933190001&cb=jsonpCallback_37&timestamp=1500714862506 HTTP/1.1","http_user_agent":"Mozilla/5.0 (Linux; Android 5.1; ZTE BA510 Build/LMY47D) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/33.0.0.0 Mobile Safari/537.36","cookie_uid":"-","size":4452,"responsetime":0.290,"upstreamtime":"0.289","upstreamhost":"192.168.10.12:8080","http_host":"c.bxb.oupeng.com","url":"/s","domain":"c.bxb.oupeng.com","xff":"-","referer":"http://www.opgirl.cn/?did=202","status":"200"}

Here is the info from kibana discover json column:

{
  "_index": "c-adbxb-cn-nginx-access-2017.07.22",
  "_type": "c-adbxb-cn-nginx-access",
  "_id": "AV1pkriwOTwuD3j9tnO5",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2017-07-22T09:13:18.554Z",
    "offset": 116378647,
    "@version": "1",
    "beat": {
      "hostname": "uy03-03",
      "name": "uy03-03",
      "version": "5.5.0"
    },
    "input_type": "log",
    "host": "uy03-03",
    "source": "/usr/local/nginx/logs/c.adbxb.cn.access.log",
    "message": "120.132.95.115 - - [22/Jul/2017:00:25:19 +0800] \"POST /c/ads/wifi HTTP/1.1\" \"\\x0A\\x10a125f4c8acbf7f2a\\x12\\x0E223.88.161.186\\x1AFDalvik/1.6.0 (Linux; U; Android 4.0.4; ZTE U795+ Build/IMM76D)(\\x022\\x07android:\\x054.0.4B\\x1C\\x08\\x01\\x10\\xE0\\x03\\x18\\xA0\\x06 \\x01(\\x002\\x03ZTE:\\x09ZTE U795+J&\\x0A\\x05A0008\\x12\\x10wifi\\xE4\\xB8\\x87\\xE8\\x83\\xBD\\xE9\\x92\\xA5\\xE5\\x8C\\x99\\x1A\\x043060*\\x05baiduR6\\x0A\\x0F868155010512989\\x12\\x11b4:98:42:e2:2c:80\\x1A\\x10ee0a80afd554b380Z7\\x0A\\x0825513984\\x10\\xD0\\x05\\x18\\xC8\\x01 \\x00(\\x000\\x038\\xAC\\x02@\\x04h\\x03h\\x06h\\xBF\\xFB\\xC2\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\x01h\\xBE\\xFB\\xC2\\xFF\\xFF\\xFF\\xFF\\xFF\\xFF\\x01p\\x09\\xBA\\x01\\x011\\xC8\\x01\\x00\" 200 29 \"-\" \"Dalvik/1.6.0 (Linux; U; Android 4.0.4; ZTE U795+ Build/IMM76D)\" - 0.006  0.002",
    "type": "c-adbxb-cn-nginx-access",
    "tags": [
      "_jsonparsefailure",
      "beats_input_codec_json_applied"
    ]
  },
  "fields": {
    "@timestamp": [
      1500714798554
    ]
  },
  "sort": [
    1500714798554
  ]
}

{
  "_index": "www-opgirl-cn-nginx-access-2017.07.22",
  "_type": "www-opgirl-cn-nginx-access",
  "_id": "AV1ppNkgOTwuD3j97yQF",
  "_version": 1,
  "_score": null,
  "_source": {
    "@timestamp": "2017-07-22T09:33:06.494Z",
    "offset": 679426855,
    "@version": "1",
    "input_type": "log",
    "beat": {
      "hostname": "uy01-04",
      "name": "uy01-04",
      "version": "5.5.0"
    },
    "host": "uy01-04",
    "source": "/usr/local/nginx/logs/www.opgirl.cn.access.log",
    "message": "182.202.168.176 - - [22/Jul/2017:06:16:45 +0800] \"GET /picture/list?gid=2885296&pl=0&strategy=1 HTTP/1.1\" \"-\" 200 683 \"http://www.opgirl.cn/?did=72\" \"Mozilla/5.0 (Linux; Android 5.1.1; vivo Y31A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/38.0.0.0 Mobile Safari/537.36 VivoBrowser/5.1.2\" - 0.002  0.002",
    "type": "www-opgirl-cn-nginx-access",
    "tags": [
      "_jsonparsefailure",
      "beats_input_codec_json_applied"
    ]
  },
  "fields": {
    "@timestamp": [
      1500715986494
    ]
  },
  "sort": [
    1500715986494
  ]
}

Is there any string can not be recognize?

I found there many errors in /var/log/logstash/logstash-plain.log:

[2017-07-22T21:28:00,201][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('.' (code 46)): Expected space separating root-level values
 at [Source: 101.227.103.243 - - [22/Jul/2017:06:08:37 +0800] "POST /b/ads HTTP/1.1" "\x0A*1500674919139_1638310_101227103243_22671_d\x12\x02\x08\x03\x1A*\x12\x17com.pajk.personaldoctor\x1A\x0F\xE5\xB9\xB3\xE5\xAE\x89\xE5\xA5\xBD\xE5\x8C\xBB\xE7\x94\x9F\x22V\x08\x02\x12\x04\x08\x01\x10\x01*\x0C223.85.218.22\x05Apple:\x07iPhone6B\x06\x08\xD0\x05\x10\x80\x0AJ$7BEC5316-0661-4CB2-965B-E943F74BCF8E`\x01*\x15\x0A\x09972676580\x12\x06\x08\x80\x05\x10\xC0\x07\x18\x01" 200 569 "-" "Jakarta Commons-HttpClient/3.1" - 0.177  0.176; line: 1, column: 9]>, :data=>"101.227.103.243 - - [22/Jul/2017:06:08:37 +0800] \"POST /b/ads HTTP/1.1\" \"\\x0A*1500674919139_1638310_101227103243_22671_d\\x12\\x02\\x08\\x03\\x1A*\\x12\\x17com.pajk.personaldoctor\\x1A\\x0F\\xE5\\xB9\\xB3\\xE5\\xAE\\x89\\xE5\\xA5\\xBD\\xE5\\x8C\\xBB\\xE7\\x94\\x9F\\x22V\\x08\\x02\\x12\\x04\\x08\\x01\\x10\\x01*\\x0C223.85.218.22\\x05Apple:\\x07iPhone6B\\x06\\x08\\xD0\\x05\\x10\\x80\\x0AJ$7BEC5316-0661-4CB2-965B-E943F74BCF8E`\\x01*\\x15\\x0A\\x09972676580\\x12\\x06\\x08\\x80\\x05\\x10\\xC0\\x07\\x18\\x01\" 200 569 \"-\" \"Jakarta Commons-HttpClient/3.1\" - 0.177  0.176"}
2

The issue is that the User-Agent field is being populated with '\x__' characters, which is the default encoding for NGINX access logs, and, unfortunately, invalid JSON. If you are using a newer version of NGINX (>=1.11.8), then you can set escape=json as an argument, which will supply a properly encoded version of the data.

See https://github.com/elastic/examples/tree/master/Common%20Data%20Formats/nginx_json_logs#warning-invalid-json for more details.

1 Like
3

Hi Rob, very useful information, thanks a lot !

4

I don't understand. The first single line example from your log looks fine and is indeed JSON, but the processed event that you've copied from Kibana clearly shows a log entry that isn't JSON and can't possibly be correctly processed by a json codec. Don't all entries in /usr/local/nginx/logs/c.adbxb.cn.access.log have the expected JSON format or what's going on?

5

There are some special characters in the user-agent field of nginx-accesslog, but they are not valid json...and not all the messages contains a special character.

6

There are some special characters in the user-agent field of nginx-accesslog, but they are not valid json...and not all the messages contains a special character.

That's not my point. According to what you posted the line from /usr/local/nginx/logs/c.adbxb.cn.access.log looks like this:

182.202.168.176 - - [22/Jul/2017:06:16:45 +0800] "GET /picture/list?gid=2885296&pl=0&strategy=1 HTTP/1.1" "-" 200 683 "http://www.opgirl.cn/?did=72" "Mozilla/5.0 (Linux; Android 5.1.1; vivo Y31A Build/LMY47V) AppleWebKit/537.36 (KHTML, like Gecko) Version/4.0 Chrome/38.0.0.0 Mobile Safari/537.36 VivoBrowser/5.1.2" - 0.002  0.002

That's not JSON at all. It's a normal Apache combined-style HTTP access log entry.

7

Yes, you attention that, I am sorry I paste the wrong message...that errors because there are a part of original logs in the logfile.

Just see the error message like this:

[2017-07-26T16:53:07,823][ERROR][logstash.codecs.json     ] JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized character escape 'x' (code 120)
 at [Source: {"@timestamp":"2017-07-26T16:53:04+08:00","host":"117.119.33.237","clientip":"106.75.21.67","remote_user":"-","request":"POST /c/ads/wifi HTTP/1.1","http_user_agent":"Dalvik/2.1.0 &#40;Linux; U; Android 5.1.1; \xC3\xA3\xC2\x80\xC2\x80\xC3\xA3\xC2\x80\xC2\x80 Build/LMY47V&#41;","cookie_uid":"-","size":7884,"responsetime":0.234,"upstreamtime":"0.208","upstreamhost":"192.168.10.16:8080","http_host":"c.adbxb.cn","url":"/c/ads/wifi","domain":"c.adbxb.cn","xff":"-","referer":"-","status":"200"}; line: 1, column: 213]>, :data=>"{\"@timestamp\":\"2017-07-26T16:53:04+08:00\",\"host\":\"117.119.33.237\",\"clientip\":\"106.75.21.67\",\"remote_user\":\"-\",\"request\":\"POST /c/ads/wifi HTTP/1.1\",\"http_user_agent\":\"Dalvik/2.1.0 &#40;Linux; U; Android 5.1.1; \\xC3\\xA3\\xC2\\x80\\xC2\\x80\\xC3\\xA3\\xC2\\x80\\xC2\\x80 Build/LMY47V&#41;\",\"cookie_uid\":\"-\",\"size\":7884,\"responsetime\":0.234,\"upstreamtime\":\"0.208\",\"upstreamhost\":\"192.168.10.16:8080\",\"http_host\":\"c.adbxb.cn\",\"url\":\"/c/ads/wifi\",\"domain\":\"c.adbxb.cn\",\"xff\":\"-\",\"referer\":\"-\",\"status\":\"200\"}"}

And the answer from Rob is right...amazing is he.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.