# Trying to work out the expected JSON format for Varnish logging
**URL:** https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185
**Category:** Beats
**Tags:** filebeat
**Created:** [February 23, 2017, 10:00am UTC](https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185 "2017-02-23T10:00:10Z")
**Posts on this page:** 6
**Page:** 1
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [February 23, 2017, 10:00am UTC](https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185/1 "2017-02-23T10:00:10Z")
I am trying to log data from Varnish, which can be coerced into producing a JSON log file.
When I import this using Filebeat, I'm getting a json\_error of
```
Error decoding JSON: invalid character '}' looking for beginning of object key string
```
An example of a log line is:
```
{"message": {"remoteHost": "5.6.7.8","remoteUser": "-","timeStamp": "2017-02-23 09:53:23","requestMethod": "GET","requestUrl": "/Scripts/93c361e6-1234-1234-1234-0d5e842ff5b8.js","queryString": "","status": 200,"timeTaken": 82,"bytes": 6541,"referrer": "https://www.example.co.uk/products/bakery","userAgent": "Mozilla/5.0 (iPad; CPU OS 9_3_5 like Mac OS X) AppleWebKit/601.1.46 (KHTML, like Gecko) Version/9.0 Mobile/13G36 Safari/601.1","xForwardedFor": "1.2.3.4","xForwardedProto": "https","destinationIp": "10.0.0.1","director": "-","xBackend": "web10","varnishHitMiss": "hit","varnishHandling": "hit","hostname": "www.mydomain.com","varnishTimeFirstByte": 0.000050545,}}
```
Although I also tried it without the outer "message" key with the same result.
This is my filebeat.yml:
```
filebeat.prospectors:
-
type: log
document_type: varnish
paths:
- /var/log/varnish/varnishncsa.log
json.message_key: "message"
json.keys_under_root: true
json.add_error_key: true
```
---
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [February 23, 2017, 10:04am UTC](https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185/2 "2017-02-23T10:04:06Z")
Aah, nevermind, I was being an idiot. I had a trailing "," at the end of the last value.
For posterity, this works as a varnishncsa format:
```
format="\"{\\\"remoteHost\\\": \\\"%h\\\",\\\"remoteUser\\\": \\\"%u\\\",\\\"timeStamp\\\": \\\"%{%Y-%m-%d %T}t\\\",\\\"requestMethod\\\": \\\"%m\\\",\\\"requestUrl\\\": \\\"%U\\\",\\\"queryString\\\": \\\"%q\\\",\\\"status\\\": %s,\\\"timeTaken\\\": %D,\\\"bytes\\\": %b,\\\"referrer\\\": \\\"%{Referer}i\\\",\\\"userAgent\\\": \\\"%{User-agent}i\\\",\\\"xForwardedFor\\\": \\\"%{X-Forwarded-For}i\\\",\\\"xForwardedProto\\\": \\\"%{X-Forwarded-Proto}i\\\",\\\"destinationIp\\\": \\\"%{dest_ip}i\\\",\\\"director\\\": \\\"%{X-Director}o\\\",\\\"xBackend\\\": \\\"%{X-Backend}o\\\",\\\"varnishHitMiss\\\": \\\"%{Varnish:hitmiss}x\\\",\\\"varnishHandling\\\": \\\"%{Varnish:handling}x\\\",\\\"hostname\\\": \\\"%{Host}i\\\",\\\"varnishTimeFirstByte\\\": %{Varnish:time_firstbyte}x}\""
```
Without a `json.message_key` setting in filebeat.yml.
---
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [February 28, 2017, 10:32am UTC](https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185/3 "2017-02-28T10:32:25Z")
I discovered I was using a bad @timestamp format, so I've fixed it here. Apologies for the insane amount of quote escaping, that's to do with how VarnishNCSA is configured 🙂
```
format="\"{\\\"remoteHost\\\": \\\"%h\\\",\\\"remoteUser\\\": \\\"%u\\\",\\\"@timestamp\\\": \\\"%{%Y-%m-%dT%TZ}t\\\",\\\"requestMethod\\\": \\\"%m\\\",\\\"requestUrl\\\": \\\"%U\\\",\\\"queryString\\\": \\\"%q\\\",\\\"status\\\": %s,\\\"timeTaken\\\": %D,\\\"bytes\\\": %b,\\\"referrer\\\": \\\"%{Referer}i\\\",\\\"userAgent\\\": \\\"%{User-agent}i\\\",\\\"xForwardedFor\\\": \\\"%{X-Forwarded-For}i\\\",\\\"xForwardedProto\\\": \\\"%{X-Forwarded-Proto}i\\\",\\\"destinationIp\\\": \\\"%{dest_ip}i\\\",\\\"director\\\": \\\"%{X-Director}o\\\",\\\"xBackend\\\": \\\"%{X-Backend}o\\\",\\\"varnishHitMiss\\\": \\\"%{Varnish:hitmiss}x\\\",\\\"varnishHandling\\\": \\\"%{Varnish:handling}x\\\",\\\"hostname\\\": \\\"%{Host}i\\\",\\\"varnishTimeFirstByte\\\": %{Varnish:time_firstbyte}x}\""
```
---
### Author:  [@ruflin](https://discuss.elastic.co/u/ruflin)
#### Post date: [March 1, 2017, 8:09am UTC](https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185/4 "2017-03-01T08:09:33Z")
Glad you found a solution. Does this also solve [Correct @timestamp format for JSON ingress](https://discuss.elastic.co/t/correct-timestamp-format-for-json-ingress/76653/1) ?
---
### Author:  [@Cylindric](https://discuss.elastic.co/u/Cylindric)
#### Post date: [March 1, 2017, 9:27am UTC](https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185/5 "2017-03-01T09:27:06Z")
Oh yes, I forgot about that. I'll post the answer in there.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 29, 2017, 9:27am UTC](https://discuss.elastic.co/t/trying-to-work-out-the-expected-json-format-for-varnish-logging/76185/6 "2017-03-29T09:27:11Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.