Trying to work out the expected JSON format for Varnish logging

[Cylindric (Mark Hanford)]

I am trying to log data from Varnish, which can be coerced into producing a JSON log file.

When I import this using Filebeat, I'm getting a json_error of

Error decoding JSON: invalid character '}' looking for beginning of object key string

An example of a log line is:

{"message": {"remoteHost": "5.6.7.8","remoteUser": "-","timeStamp": "2017-02-23 09:53:23","requestMethod": "GET","requestUrl": "/Scripts/93c361e6-1234-1234-1234-0d5e842ff5b8.js","queryString": "","status": 200,"timeTaken": 82,"bytes": 6541,"referrer": "https://www.example.co.uk/products/bakery","userAgent": "Mozilla/5.0 (iPad; CPU OS 9_3_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13G36 Safari/601.1","xForwardedFor": "1.2.3.4","xForwardedProto": "https","destinationIp": "10.0.0.1","director": "-","xBackend": "web10","varnishHitMiss": "hit","varnishHandling": "hit","hostname": "www.mydomain.com","varnishTimeFirstByte": 0.000050545,}}

Although I also tried it without the outer "message" key with the same result.

This is my filebeat.yml:

filebeat.prospectors:
-
  type: log
  document_type: varnish
  paths:
    - /var/log/varnish/varnishncsa.log
  json.message_key: "message"
  json.keys_under_root: true
  json.add_error_key: true

[Cylindric (Mark Hanford)]

Aah, nevermind, I was being an idiot. I had a trailing "," at the end of the last value.

For posterity, this works as a varnishncsa format:

format="\"{\\\"remoteHost\\\": \\\"%h\\\",\\\"remoteUser\\\": \\\"%u\\\",\\\"timeStamp\\\": \\\"%{%Y-%m-%d %T}t\\\",\\\"requestMethod\\\": \\\"%m\\\",\\\"requestUrl\\\": \\\"%U\\\",\\\"queryString\\\": \\\"%q\\\",\\\"status\\\": %s,\\\"timeTaken\\\": %D,\\\"bytes\\\": %b,\\\"referrer\\\": \\\"%{Referer}i\\\",\\\"userAgent\\\": \\\"%{User-agent}i\\\",\\\"xForwardedFor\\\": \\\"%{X-Forwarded-For}i\\\",\\\"xForwardedProto\\\": \\\"%{X-Forwarded-Proto}i\\\",\\\"destinationIp\\\": \\\"%{dest_ip}i\\\",\\\"director\\\": \\\"%{X-Director}o\\\",\\\"xBackend\\\": \\\"%{X-Backend}o\\\",\\\"varnishHitMiss\\\": \\\"%{Varnish:hitmiss}x\\\",\\\"varnishHandling\\\": \\\"%{Varnish:handling}x\\\",\\\"hostname\\\": \\\"%{Host}i\\\",\\\"varnishTimeFirstByte\\\": %{Varnish:time_firstbyte}x}\""

Without a json.message_key setting in filebeat.yml.

[Cylindric (Mark Hanford)]

I discovered I was using a bad @timestamp format, so I've fixed it here. Apologies for the insane amount of quote escaping, that's to do with how VarnishNCSA is configured

format="\"{\\\"remoteHost\\\": \\\"%h\\\",\\\"remoteUser\\\": \\\"%u\\\",\\\"@timestamp\\\": \\\"%{%Y-%m-%dT%TZ}t\\\",\\\"requestMethod\\\": \\\"%m\\\",\\\"requestUrl\\\": \\\"%U\\\",\\\"queryString\\\": \\\"%q\\\",\\\"status\\\": %s,\\\"timeTaken\\\": %D,\\\"bytes\\\": %b,\\\"referrer\\\": \\\"%{Referer}i\\\",\\\"userAgent\\\": \\\"%{User-agent}i\\\",\\\"xForwardedFor\\\": \\\"%{X-Forwarded-For}i\\\",\\\"xForwardedProto\\\": \\\"%{X-Forwarded-Proto}i\\\",\\\"destinationIp\\\": \\\"%{dest_ip}i\\\",\\\"director\\\": \\\"%{X-Director}o\\\",\\\"xBackend\\\": \\\"%{X-Backend}o\\\",\\\"varnishHitMiss\\\": \\\"%{Varnish:hitmiss}x\\\",\\\"varnishHandling\\\": \\\"%{Varnish:handling}x\\\",\\\"hostname\\\": \\\"%{Host}i\\\",\\\"varnishTimeFirstByte\\\": %{Varnish:time_firstbyte}x}\""

[ruflin]

Glad you found a solution. Does this also solve Correct @timestamp format for JSON ingress ?

[Cylindric (Mark Hanford)]

Oh yes, I forgot about that. I'll post the answer in there.