# Parse JSON data with filebeat
**URL:** https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008
**Category:** Beats
**Tags:** filebeat
**Created:** [March 25, 2017, 8:08pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008 "2017-03-25T20:08:24Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@calmandniceperson](https://discuss.elastic.co/u/calmandniceperson)
#### Post date: [March 25, 2017, 8:08pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/1 "2017-03-25T20:08:24Z")
I'm trying to parse JSON logs our server application is producing. It's writing to 3 log files in a directory I'm mounting in a Docker container running Filebeat. So far so good, it's reading the log files all right. However, in Kibana, the messages arrive, but the content itself it just shown as a field called "message" and the data in the content field is not accessible via its own fields (like "source\_address", etc.).
I'm writing the logs using [logrus](https://github.com/Sirupsen/logrus) and I want Beats to pass them straight on to Elasticsearch. I went through all kinds of documentation and found multiple ways of achieving what I want, but none of it's working.
This is my current configuration for filebeat:
```
filebeat.prospectors:
- paths:
- /mnt/log/*.log
input_type: log
multiline.pattern: '^{'
multiline.negate: false
multiline.match: after
processors:
- decode_json_fields:
fields: ['message']
target: json
output.elasticsearch:
hosts: ["elasticsearch:9200"]
template.name: filebeat
template.path: filebeat.template.json
```
A little nudge in the right direction would be greatly appreciated.
---
### Author:  [@tudor](https://discuss.elastic.co/u/tudor)
#### Post date: [March 27, 2017, 10:57am UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/2 "2017-03-27T10:57:33Z")
Are the JSON objects written as one per line? The multiline configuration seems to indicate that are not. In general, I'd recommend:
- configuring logrus to output one object per line
- remove the multiline configuration
- Use the [json decoding](https://www.elastic.co/guide/en/beats/filebeat/5.2/configuration-filebeat-options.html#config-json) at the prospector level, instead of the processor
See, for example, this [blog post](https://www.elastic.co/blog/structured-logging-filebeat) on the topic.
---
### Author:  [@calmandniceperson](https://discuss.elastic.co/u/calmandniceperson)
#### Post date: [March 27, 2017, 4:47pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/3 "2017-03-27T16:47:17Z")
Thank you for taking the time to respond.
I changed the configuration of Beats to this:
```
filebeat.prospectors:
- paths:
- /mnt/log/*.log
input_type: log
json.keys_under_root: true
json.add_error_key: true
json.message_key: log
output.elasticsearch:
hosts: ["elasticsearch:9200"]
template.name: filebeat
template.path: filebeat.template.json
```
This is what the log files look like. Every log is on its own line:
```
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-25T14:57:15Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-25T14:57:18Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-27T16:36:49Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-27T16:36:52Z"}
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-27T16:36:55Z"}
```
Yet still, Kibana only shows a "message" field. I have a strange feeling that I'm using the message\_key field in a wrong way. Am I using it right? What do I use as the key?
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [March 27, 2017, 5:14pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/4 "2017-03-27T17:14:43Z")
For those logs I would use `message_key: msg`. But you can also omit that config option because it's optional and you are not doing anything that would required it.
I ran a quick test with your logs and this config and it worked as expected.
```auto
filebeat.prospectors:
- paths:
- test.json
input_type: log
json.keys_under_root: true
json.add_error_key: true
output.elasticsearch:
hosts: ["http://localhost:9200"]
```
I was getting data like:
```auto
{
"@timestamp": "2017-03-27T17:11:31.359Z",
"beat": {
"hostname": "x",
"name": "x",
},
"error": "dial tcp 172.21.0.2:5432: getsockopt: connection refused",
"input_type": "log",
"level": "error",
"msg": "Could not open database connection",
"offset": 790,
"source": "/Users/akroh/go/src/github.com/elastic/beats/filebeat/.test/logrus/test.json",
"time": "2017-03-27T16:36:55Z",
"type": "log"
}
```
---
### Author:  [@calmandniceperson](https://discuss.elastic.co/u/calmandniceperson)
#### Post date: [March 27, 2017, 5:52pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/5 "2017-03-27T17:52:44Z")
I queried Elasticsearch and got the following:
```
{
"_index" : "filebeat-2017.03.27",
"_type" : "log",
"_id" : "AVsQ0S06i5178hY1Lr4U",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2017-03-27T17:29:16.356Z",
"beat" : {
"hostname" : "15f7ccc3a4c1",
"name" : "15f7ccc3a4c1",
"version" : "5.2.2"
},
"input_type" : "log",
"message" : "{\"error\":\"dial tcp 172.21.0.2:5432: getsockopt: connection refused\",\"level\":\"error\",\"msg\":\"Could not open database connection\",\"time\":\"2017-03-27T17:19:35Z\"}",
"offset" : 158,
"source" : "/mnt/log/error.log",
"type" : "log"
}
}
```
Could it somehow be related to Docker? I recreated all containers before starting the server and checking the logs. I suppose it doesn't matter which file endings the log files have (.json or .log)?
---
### Author:  [@calmandniceperson](https://discuss.elastic.co/u/calmandniceperson)
#### Post date: [March 27, 2017, 7:14pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/6 "2017-03-27T19:14:25Z")
I just tested changing the configuration to this:
```
filebeat.prospectors:
- paths:
- /mnt/log/info.json
input_type: log
json.keys_under_root: true
json.add_error_key: true
output.elasticsearch:
hosts: ["elasticsearch:9200"]
```
What happens now is that filebeat still reads my \*.log files, but doesn't read the .json file specified in the config.
This is the output I get when querying Elasticsearch:
```
{
"took" : 3,
"timed_out" : false,
"_shards" : {
"total" : 6,
"successful" : 6,
"failed" : 0
},
"hits" : {
"total" : 3,
"max_score" : 1.0,
"hits" : [
{
"_index" : ".kibana",
"_type" : "config",
"_id" : "5.2.2",
"_score" : 1.0,
"_source" : {
"buildNum" : 14723
}
},
{
"_index" : "filebeat-2017.03.27",
"_type" : "log",
"_id" : "AVsRLezyYgiGm0kvKUi9",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2017-03-27T19:10:36.110Z",
"beat" : {
"hostname" : "8deef5bae0db",
"name" : "8deef5bae0db",
"version" : "5.2.2"
},
"input_type" : "log",
"message" : "{\"error\":\"dial tcp 172.21.0.3:5432: getsockopt: connection refused\",\"level\":\"error\",\"msg\":\"Could not open database connection\",\"time\":\"2017-03-27T19:10:35Z\"}",
"offset" : 158,
"source" : "/mnt/log/error.log",
"type" : "log"
}
},
{
"_index" : "filebeat-2017.03.27",
"_type" : "log",
"_id" : "AVsRLezyYgiGm0kvKUi-",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2017-03-27T19:10:39.111Z",
"beat" : {
"hostname" : "8deef5bae0db",
"name" : "8deef5bae0db",
"version" : "5.2.2"
},
"input_type" : "log",
"message" : "{\"error\":\"dial tcp 172.21.0.3:5432: getsockopt: connection refused\",\"level\":\"error\",\"msg\":\"Could not open database connection\",\"time\":\"2017-03-27T19:10:38Z\"}",
"offset" : 316,
"source" : "/mnt/log/error.log",
"type" : "log"
}
}
]
}
}
```
---
### Author:  [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)
#### Post date: [March 27, 2017, 7:19pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/7 "2017-03-27T19:19:53Z")
It sounds like a configuration issue with your Filebeat container where it's not using the config file you expected. Could you trying running Filebeat outside of docker and see if you can reproduce the issue.
File extensions don't matter to Filebeat.
---
### Author:  [@calmandniceperson](https://discuss.elastic.co/u/calmandniceperson)
#### Post date: [March 27, 2017, 7:34pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/8 "2017-03-27T19:34:16Z")
I'm so sorry for wasting your time. What you just wrote made me check the documentation of the official beats docker image again. I put the configuration file in the wrong directory. It's now working as intended.
```
{
"_index" : "filebeat-2017.03.27",
"_type" : "log",
"_id" : "AVsRP1CAXkvX0lRuH3-Y",
"_score" : 1.0,
"_source" : {
"@timestamp" : "2017-03-27T19:30:01.146Z",
"beat" : {
"hostname" : "1a38663b0526",
"name" : "1a38663b0526",
"version" : "5.2.2"
},
"input_type" : "log",
"level" : "info",
"msg" : "Directory created",
"offset" : 306,
"path" : "/toronto/media/",
"source" : "/mnt/log/info.log",
"time" : "2017-03-27T19:29:53Z",
"type" : "log"
}
```
Thank you very much for your help! @andrewkroh @tudor
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [April 24, 2017, 7:34pm UTC](https://discuss.elastic.co/t/parse-json-data-with-filebeat/80008/9 "2017-04-24T19:34:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.