# Can i read a formatted json file via filebeat?

**URL:** <https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015>\
**Category:** Beats\
**Created:** [December 16, 2018, 6:58am UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015 "2018-12-16T06:58:06Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![fillic2002](https://avatars.discourse-cdn.com/v4/letter/f/c2a13f/32.png) [@fillic2002](https://discuss.elastic.co/u/fillic2002)\
**Post date:** [December 16, 2018, 6:58am UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/1 "2018-12-16T06:58:07Z")

</div>

I see the beat is lightweight and don't want to go through logstash heavy processing pipeline. Is there a way i can use any beat to parse my json file and parse individual field as part of index column? i am reading many question but seems like none is pointing to my requirement.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 16, 2018, 8:41am UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/2 "2018-12-16T08:41:50Z")

</div>

One of the reasons Beats are “light-weight” is that they offer a lot less flexibility around processing data. You can however use them together with ingest node pipelines, which may support the processing you need.

---

<div class="post-metadata">

**Author:** ![fillic2002](https://avatars.discourse-cdn.com/v4/letter/f/c2a13f/32.png) [@fillic2002](https://discuss.elastic.co/u/fillic2002)\
**Post date:** [December 16, 2018, 10:59am UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/3 "2018-12-16T10:59:20Z")

</div>

You mean to use plugin directly for elasticsearch?  
[https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html](https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html)

What if my output of beat is not ES cluster? in that scenario what would be the best way to read JSON?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 16, 2018, 11:24am UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/4 "2018-12-16T11:24:26Z")

</div>

No, I mean [ingest node](https://www.elastic.co/blog/should-i-use-logstash-or-elasticsearch-ingest-nodes). [Filebeat can parse JSON](https://www.elastic.co/guide/en/beats/filebeat/6.5/decode-json-fields.html), so if you are not enriching or transforming the data that may work as well. There is however a limited number of outputs supported, which may limit where you can send data.

It would probably be a lot easier if you could show the data and describe exactly what you want to do with it and where you want to send it.

---

<div class="post-metadata">

**Author:** ![fillic2002](https://avatars.discourse-cdn.com/v4/letter/f/c2a13f/32.png) [@fillic2002](https://discuss.elastic.co/u/fillic2002)\
**Post date:** [December 17, 2018, 1:05pm UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/5 "2018-12-17T13:05:40Z")

</div>

[{  
"connector\_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",  
"group\_guids": ["08fba752-6034-4a38-985a-4819b54b136b"],  
"id": 6.419229331435815e+18,  
"date": "2018-10-01T13:41:05+00:00",  
"file": {  
"disposition": "Malicious",  
"identity": {  
"sha256": "ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa"  
}  
},  
"event\_type\_id": 5.53648143e+08,  
"computer": {  
"network\_addresses": [{  
"mac": "x:x:x:x:x",  
"ip": "1.1.1.1"  
}],  
"links": {  
"computer": "[https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb](https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb)",  
"trajectory": "[https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory](https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory)",  
"group": "[https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b](https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b)"  
},  
"connector\_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",  
"hostname": "Demo\_WannaCry\_Ransomware",  
"external\_ip": "171.190.158.103",  
"active": true  
},  
"timestamp": 1.538401265e+09,  
"detection\_id": "6419229327140847660",  
"timestamp\_nanoseconds": 1.66e+08,  
"event\_type": "Threat Quarantined"  
},  
{  
"event\_type\_id": 2.16426088e+09,  
"computer": {  
"external\_ip": "171.190.158.103",  
"active": true,  
"network\_addresses": [{  
"ip": "2.2.2.2",  
"mac": "y:y:y:y"  
}],  
"links": {  
"trajectory": "[https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory](https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory)",  
"group": "[https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b](https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b)",  
"computer": "[https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb](https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb)"  
},  
"connector\_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",  
"hostname": "Demo\_WannaCry\_Ransomware"  
},  
"file": {  
"disposition": "Malicious",  
"identity": {  
"sha256": "ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa"  
}  
},  
"event\_type": "Quarantine Failure",  
"timestamp\_nanoseconds": 1.66e+08,  
"detection\_id": "6419229327140847659",  
"connector\_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",  
"id": 6.419229331435815e+18,  
"group\_guids": ["08fba752-6034-4a38-985a-4819b54b136b"],  
"error": {  
"description": "Cannot delete",  
"error\_code": 3.221225761e+09  
},  
"timestamp": 1.538401265e+09,  
"date": "2018-10-01T13:41:05+00:00"  
},  
{  
"group\_guids": ["08fba752-6034-4a38-985a-4819b54b136b"],  
"event\_type\_id": 1.090519054e+09,  
"detection\_id": "6419229327140847667",  
"connector\_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",  
"date": "2018-10-01T13:41:04+00:00",  
"event\_type": "Threat Detected",  
"detection": "W32.File.MalParent",  
"computer": {  
"hostname": "Demo\_WannaCry\_Ransomware",  
"external\_ip": "171.190.158.103",  
"user": "",  
"active": true,  
"network\_addresses": [{  
"ip": "3.3.3.3",  
"mac": "z:z:z:z"  
}],  
"links": {  
"computer": "[https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb](https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb)",  
"trajectory": "[https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory](https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory)",  
"group": "[https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b](https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b)"  
},  
"connector\_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb"  
},  
"id": 6.419229327140848e+18,  
"timestamp": 1.538401264e+09,  
"timestamp\_nanoseconds": 7.3e+08,  
"file": {  
"disposition": "Malicious",  
"file\_name": "tasksche.exe",  
"file\_path": "\\?\C:\ProgramData\qzkbplcgew884\tasksche.exe",  
"identity": {  
"sha256": "ed01ebfbc9eb5bbea545af4d01bf5f10716618404a80439c6e5babe8e080e41aa",  
"sha1": "5ff465afaabcbf0150d1a3ab2c2e74f3a4426467",  
"md5": "84c82835a5d21bbcf75a61706d8ab549"  
}  
}  
}]

---

<div class="post-metadata">

**Author:** ![fillic2002](https://avatars.discourse-cdn.com/v4/letter/f/c2a13f/32.png) [@fillic2002](https://discuss.elastic.co/u/fillic2002)\
**Post date:** [December 17, 2018, 1:07pm UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/6 "2018-12-17T13:07:08Z")

</div>

above mentioned is my json which i need to read. I tried option "json.keys\_under\_root: true", which works well if the json is single line object. but that is not working on above mentioned json.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [December 17, 2018, 1:21pm UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/7 "2018-12-17T13:21:06Z")

</div>

That is a list of JSON objects, which you will need to break into separate events. For that you will need more advanced processing than Filebeat is capable of. You may need to use Logstash here as I am not sure ingest node pipelines support splitting an event into multiple ones.

---

<div class="post-metadata">

**Author:** ![fillic2002](https://avatars.discourse-cdn.com/v4/letter/f/c2a13f/32.png) [@fillic2002](https://discuss.elastic.co/u/fillic2002)\
**Post date:** [December 17, 2018, 1:38pm UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/8 "2018-12-17T13:38:29Z")

</div>

if i break each list item in a single line json objects, Should that work? as i tried that but no luck

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 14, 2019, 3:38pm UTC](https://discuss.elastic.co/t/can-i-read-a-formatted-json-file-via-filebeat/161015/9 "2019-01-14T15:38:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.