Filebeat doing wrong jsonification

[vaidabhishek (Abhishek Vaid)]

• I'm using filebeat to send logs from Nginx to ES. Strangely enough, I found this document in my ES.

{
    "@timestamp": "2018-03-30T12:28:52.396Z",
    "source": "/var/log/nginx/access_upstream.log",
    "offset": 3118732890,
    "message": "{\"created_at\": \"2018-03-30T12:28:52+00:00\", \"remote_addr\": \"xx.xx.xx.xx\", \"remote_user\": \"-\", \"request\": \"GET /hyderabad-fitness-trainer-at-home-langar-house-north-golconda-fort?p=10 HTTP/1.1\", \"request_method\": \"GET\", \"postdata\" : \"-\", \"request_response_time\": 0.183, \"upstream_response_time\": 0.183, \"upstream_addr\": yy.yy.yy.yy:80, \"body_bytes_sent\": 58111, \"msec\": 1522412932.224, \"status\": 404, \"http_referrer\": \"-\", \"platform\": \"desktop\", \"chcount\": \"0\", \"device\": \"desktop\", \"http_user_agent\": \"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)\"}",
    "prospector": {
      "type": "log"
    },
    "beat": {
      "hostname": "xxxxx",
      "version": "6.2.0",
      "name": "xxxx"
    }
  }

• Nothing seems out of ordinary at cursory look. But digging deeply, you can see the one of the values upstream_addr has a non-json value (neither Number or String). Oddly enough, the value of another field remote_addr is made string.

• I realized this when I tried to convert this json string to dictionary in pythonic code. Something clearly seems broken here.

[pierhugues (Pier-Hugues Pellerin)]

Hello @vaidabhishek You are correct upstream_addr is invalid JSON, it should be a string. But from what I see with your output, you are using the default configuration and you are sending directly the content that Filebeat read from the file to Elasticsearch.

In short, Filebeat read the file one line at a time and send the content to ES, for Filebeat this is just a string and not JSON.

I suspect the line is problematic in the original file, can you check that? If its the case its probably a problem with you logging library. There are ways to fix it after, but it is better to fix it at the source.