Cylindric
(Mark Hanford)
February 23, 2017, 10:00am
1
I am trying to log data from Varnish, which can be coerced into producing a JSON log file.
When I import this using Filebeat, I'm getting a json_error of
Error decoding JSON: invalid character '}' looking for beginning of object key string
An example of a log line is:
{"message": {"remoteHost": "5.6.7.8","remoteUser": "-","timeStamp": "2017-02-23 09:53:23","requestMethod": "GET","requestUrl": "/Scripts/93c361e6-1234-1234-1234-0d5e842ff5b8.js","queryString": "","status": 200,"timeTaken": 82,"bytes": 6541,"referrer": "https://www.example.co.uk/products/bakery","userAgent": "Mozilla/5.0 (iPad; CPU OS 9_3_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13G36 Safari/601.1","xForwardedFor": "1.2.3.4","xForwardedProto": "https","destinationIp": "10.0.0.1","director": "-","xBackend": "web10","varnishHitMiss": "hit","varnishHandling": "hit","hostname": "www.mydomain.com","varnishTimeFirstByte": 0.000050545,}}
Although I also tried it without the outer "message" key with the same result.
This is my filebeat.yml:
filebeat.prospectors:
-
type: log
document_type: varnish
paths:
- /var/log/varnish/varnishncsa.log
json.message_key: "message"
json.keys_under_root: true
json.add_error_key: true
Cylindric
(Mark Hanford)
February 23, 2017, 10:04am
2
Aah, nevermind, I was being an idiot. I had a trailing "," at the end of the last value.
For posterity, this works as a varnishncsa format:
format="\"{\\\"remoteHost\\\": \\\"%h\\\",\\\"remoteUser\\\": \\\"%u\\\",\\\"timeStamp\\\": \\\"%{%Y-%m-%d %T}t\\\",\\\"requestMethod\\\": \\\"%m\\\",\\\"requestUrl\\\": \\\"%U\\\",\\\"queryString\\\": \\\"%q\\\",\\\"status\\\": %s,\\\"timeTaken\\\": %D,\\\"bytes\\\": %b,\\\"referrer\\\": \\\"%{Referer}i\\\",\\\"userAgent\\\": \\\"%{User-agent}i\\\",\\\"xForwardedFor\\\": \\\"%{X-Forwarded-For}i\\\",\\\"xForwardedProto\\\": \\\"%{X-Forwarded-Proto}i\\\",\\\"destinationIp\\\": \\\"%{dest_ip}i\\\",\\\"director\\\": \\\"%{X-Director}o\\\",\\\"xBackend\\\": \\\"%{X-Backend}o\\\",\\\"varnishHitMiss\\\": \\\"%{Varnish:hitmiss}x\\\",\\\"varnishHandling\\\": \\\"%{Varnish:handling}x\\\",\\\"hostname\\\": \\\"%{Host}i\\\",\\\"varnishTimeFirstByte\\\": %{Varnish:time_firstbyte}x}\""
Without a json.message_key
setting in filebeat.yml.
Cylindric
(Mark Hanford)
February 28, 2017, 10:32am
3
I discovered I was using a bad @timestamp format, so I've fixed it here. Apologies for the insane amount of quote escaping, that's to do with how VarnishNCSA is configured
format="\"{\\\"remoteHost\\\": \\\"%h\\\",\\\"remoteUser\\\": \\\"%u\\\",\\\"@timestamp\\\": \\\"%{%Y-%m-%dT%TZ}t\\\",\\\"requestMethod\\\": \\\"%m\\\",\\\"requestUrl\\\": \\\"%U\\\",\\\"queryString\\\": \\\"%q\\\",\\\"status\\\": %s,\\\"timeTaken\\\": %D,\\\"bytes\\\": %b,\\\"referrer\\\": \\\"%{Referer}i\\\",\\\"userAgent\\\": \\\"%{User-agent}i\\\",\\\"xForwardedFor\\\": \\\"%{X-Forwarded-For}i\\\",\\\"xForwardedProto\\\": \\\"%{X-Forwarded-Proto}i\\\",\\\"destinationIp\\\": \\\"%{dest_ip}i\\\",\\\"director\\\": \\\"%{X-Director}o\\\",\\\"xBackend\\\": \\\"%{X-Backend}o\\\",\\\"varnishHitMiss\\\": \\\"%{Varnish:hitmiss}x\\\",\\\"varnishHandling\\\": \\\"%{Varnish:handling}x\\\",\\\"hostname\\\": \\\"%{Host}i\\\",\\\"varnishTimeFirstByte\\\": %{Varnish:time_firstbyte}x}\""
ruflin
(ruflin)
March 1, 2017, 8:09am
4
Glad you found a solution. Does this also solve Correct @timestamp format for JSON ingress ?
Cylindric
(Mark Hanford)
March 1, 2017, 9:27am
5
Oh yes, I forgot about that. I'll post the answer in there.
system
(system)
Closed
March 29, 2017, 9:27am
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.