# Filebeat parse json
**URL:** https://discuss.elastic.co/t/filebeat-parse-json/130834
**Category:** Beats
**Tags:** filebeat
**Created:** [May 7, 2018, 12:32pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834 "2018-05-07T12:32:57Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)
#### Post date: [May 7, 2018, 12:32pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/1 "2018-05-07T12:32:57Z")
Filebeat 6.2.1
I don't think filebeat is seeing my json document inside of an array `[]`. This causes json parse failures. Is there something I should be doing different to process this document?
Original document:
```auto
[{"traceID":9261530138935300,"messageType":"Default","eventType":"logEvent","signpostID":0,"source":null,"activityIdentifier":0,"subsystem":"com.apple.sharing","category":"AirDrop","threadID":6008981,"senderImageUUID":"34BD8CEF-7770-3DF6-9B02-B32788EA0C35","processImagePath":"/usr/libexec/sharingd","senderImagePath":"/usr/libexec/sharingd","timestamp":"2018-05-07 08:04:36.381104-0400","machTimestamp":1557005637979752,"eventMessage":"startSending, validated airdrop items. properties: {\n ConvertMediaFormats = 0;\n Files = (\n {\n ConvertMediaFormats = 0;\n FileBomPath = \"./lunch.pdf\";\n FileIsDirectory = 0;\n FileName = \"lunch.pdf\";\n FileType = \"com.adobe.pdf\";\n }\n );\n ReceiverComputerName = \"Tim\\U2019s iPhone\";\n ReceiverID = daffcc6a517d;\n VerifiableIdentity = 0;\n}","processImageUUID":"34BD8CEF-7770-3DF6-9B02-B32788EA0C35","processID":45882,"senderProgramCounter":1086398,"parentActivityIdentifier":0,"timezoneName":""}]
```
Filebeat.yml:
```auto
filebeat.prospectors:
- type: log
enabled: true
paths:
- /path/to/airdrop.json
json.keys_under_root: true
json.add_error_key: true
tags: ["airdrop"]
processors:
- drop_fields:
fields: ["beat.name", "beat.hostname", "beat.version", "beat", "host", "input_type", "source", "prospector.type"]
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 8
output.console:
pretty: true
```
Output from console:
```auto
{
"@timestamp": "2018-05-07T12:23:51.899Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "6.2.1"
},
"tags": [
"airdrop"
],
"prospector": {},
"offset": 1048,
"error": {
"type": "json",
"message": "Error decoding JSON: json: cannot unmarshal array into Go value of type map[string]interface {}"
}
}
```
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [May 7, 2018, 2:44pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/2 "2018-05-07T14:44:03Z")
Currently, filebeat expects one json object per-line and not an array. In the future we might overcome this limitation.
If you don't have control over the log format, you still can get rid of JSON settings under the log prospector.
Remove this lines:
```auto
json.keys_under_root: true
json.add_error_key: true
```
And have the `decode_json_fields` prospector parse the log.
This is the output I get:
```auto
{
"@timestamp": "2018-05-07T14:43:41.233Z",
"@metadata": {
"beat": "filebeat",
"type": "doc",
"version": "7.0.0-alpha1"
},
"offset": 1048,
"message": [
{
"signpostID": "0",
"subsystem": "com.apple.sharing",
"category": "AirDrop",
"threadID": "6008981",
"activityIdentifier": "0",
"parentActivityIdentifier": "0",
"timezoneName": "",
"traceID": "9261530138935300",
"senderImageUUID": "34BD8CEF-7770-3DF6-9B02-B32788EA0C35",
"senderImagePath": "/usr/libexec/sharingd",
"timestamp": "2018-05-07 08:04:36.381104-0400",
"machTimestamp": "1557005637979752",
"eventMessage": "startSending, validated airdrop items. properties: {\n ConvertMediaFormats = 0;\n Files = (\n {\n ConvertMediaFormats = 0;\n FileBomPath = \"./lunch.pdf\";\n FileIsDirectory = 0;\n FileName = \"lunch.pdf\";\n FileType = \"com.adobe.pdf\";\n }\n );\n ReceiverComputerName = \"Tim\\U2019s iPhone\";\n ReceiverID = daffcc6a517d;\n VerifiableIdentity = 0;\n}",
"processID": "45882",
"senderProgramCounter": "1086398",
"messageType": "Default",
"eventType": "logEvent",
"source": null,
"processImagePath": "/usr/libexec/sharingd",
"processImageUUID": "34BD8CEF-7770-3DF6-9B02-B32788EA0C35"
}
],
"tags": [
"airdrop"
],
"prospector": {},
"input": {
"type": "log"
}
}
```
---
### Author:  [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)
#### Post date: [May 7, 2018, 2:53pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/3 "2018-05-07T14:53:52Z")
What is the `decode_json_fields.process_array` for? I thought it would be to process arrays in json data?
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [May 7, 2018, 3:04pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/4 "2018-05-07T15:04:18Z")
There are two separate facilities at work here.
One is the `log` prospector json support, which does not support arrays.
Another one is the `decode_json_fields` processor. This one does support arrays if the `process_array` flag is set.
The main difference in your case is that `decode_jon_fields` you cannot use the `fields_under_root` functionality.
---
### Author:  [@tgdesrochers](https://discuss.elastic.co/u/tgdesrochers)
#### Post date: [May 7, 2018, 3:06pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/5 "2018-05-07T15:06:16Z")
Thank you. I did as you stated and it worked. so at least I can process the logs. Thanks for the help. I can fix the rest of the issues with `\n` and `=` to create valid json. Then all sould be well.
Thank you for the quick response
---
### Author:  [@pbala2018](https://discuss.elastic.co/u/pbala2018)
#### Post date: [May 8, 2018, 8:56pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/6 "2018-05-08T20:56:40Z")
Hi, i am new to filebeat, When i tried with above filbeat.yml, i am having an issue on publishing console output/ to logstash but file beat processing the logs, please find the log below
2018-05-08T22:46:43.923+0200 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.1
2018-05-08T22:46:43.925+0200 INFO instance/beat.go:301 filebeat start running.
2018-05-08T22:46:43.925+0200 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-05-08T22:46:43.925+0200 INFO registrar/registrar.go:108 Loading registrar data from D:\filebeat-6.2.1-windows-x86\_64\data\registry
2018-05-08T22:46:43.927+0200 INFO registrar/registrar.go:119 States Loaded from registrar: 1
2018-05-08T22:46:43.928+0200 WARN beater/filebeat.go:261 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-05-08T22:46:43.928+0200 INFO crawler/crawler.go:48 Loading Prospectors: 1
2018-05-08T22:46:43.928+0200 INFO log/prospector.go:111 Configured paths: [D:\VODMonitoring\post-data\a515734e-b97f-4787-a0d8-5d541e859295.json]
2018-05-08T22:46:43.928+0200 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 1
2018-05-08T22:46:43.929+0200 INFO log/harvester.go:216 Harvester started for file: D:\VODMonitoring\post-data\a515734e-b97f-4787-a0d8-5d541e859295.json
2018-05-08T22:47:14.025+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral\_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":30137}},"memstats":{"gc\_next":4194304,"memory\_alloc":1557480,"memory\_total":3103296,"rss":16474112}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open\_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":2,"update":1},"writes":1},"system":{"cpu":{"cores":12}}}}}
2018-05-08T22:47:43.927+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral\_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":60039}},"memstats":{"gc\_next":4194304,"memory\_alloc":1622152,"memory\_total":3167968,"rss":2084864}},"filebeat":{"harvester":{"open\_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2018-05-08T22:48:13.927+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral\_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":90039}},"memstats":{"gc\_next":4194304,"memory\_alloc":1685224,"memory\_total":3231040,"rss":32768}},"filebeat":{"harvester":{"open\_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2018-05-08T22:48:43.926+0200 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":140,"time":140},"total":{"ticks":171,"time":171,"value":171},"user":{"ticks":31,"time":31}},"info":{"ephemeral\_id":"f25c49fe-19e3-4095-ae85-02aed7bd5a16","uptime":{"ms":120038}},"memstats":{"gc\_next":4194304,"memory\_alloc":1743624,"memory\_total":3289440,"rss":-8192}},"filebeat":{"harvester":{"open\_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2018-05-08T22:48:46.736+0200 INFO beater/filebeat.go:323 Stopping filebeat
2018-05-08T22:48:46.745+0200 INFO crawler/crawler.go:109 Stopping Crawler
2018-05-08T22:48:46.745+0200 INFO crawler/crawler.go:119 Stopping 1 prospectors
2018-05-08T22:48:46.745+0200 INFO prospector/prospector.go:121 Prospector ticker stopped
2018-05-08T22:48:46.745+0200 INFO prospector/prospector.go:138 Stopping Prospector: 5865738280652087874
2018-05-08T22:48:46.745+0200 INFO log/harvester.go:237 Reader was closed: D:\VODMonitoring\post-data\a515734e-b97f-4787-a0d8-5d541e859295.json. Closing.
2018-05-08T22:48:46.745+0200 INFO crawler/crawler.go:135 Crawler stopped
2018-05-08T22:48:46.745+0200 INFO registrar/registrar.go:210 Stopping Registrar
2018-05-08T22:48:46.745+0200 INFO registrar/registrar.go:165 Ending Registrar
2018-05-08T22:48:46.746+0200 INFO instance/beat.go:308 filebeat stopped.
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [May 9, 2018, 3:36pm UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/7 "2018-05-09T15:36:26Z")
Hi @pbala2018
It's better if you open a separate thread instead of reusing this one.
---
### Author:  [@pbala2018](https://discuss.elastic.co/u/pbala2018)
#### Post date: [May 10, 2018, 6:47am UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/8 "2018-05-10T06:47:12Z")
Thanks for update @adrisr. i have created separate thread
> [@Filebeat 6.2.4 not processing JSON log and not submitting to Logstsash](https://discuss.elastic.co/t/filebeat-6-2-4-not-processing-json-log-and-not-submitting-to-logstsash/130715):
>
> Hi, I am trying to process a simple JSON file into Logstash but i see the logstash which is listening at port 5045 is not receiving any data from filebeat. Here is my filebeat and logstash config #=========================== Filebeat prospectors ============================= filebeat.prospectors: type: log enabled: true paths: - D:\ELK\post-data\*.json document\_type: json json.keys\_under\_root: true json.add\_error\_key: true processors: decode\_json\_fields: fields: ['message'] target…
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 7, 2018, 6:47am UTC](https://discuss.elastic.co/t/filebeat-parse-json/130834/9 "2018-06-07T06:47:19Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.