I see the beat is lightweight and don't want to go through logstash heavy processing pipeline. Is there a way i can use any beat to parse my json file and parse individual field as part of index column? i am reading many question but seems like none is pointing to my requirement.
One of the reasons Beats are “light-weight” is that they offer a lot less flexibility around processing data. You can however use them together with ingest node pipelines, which may support the processing you need.
You mean to use plugin directly for elasticsearch?
https://www.elastic.co/guide/en/elasticsearch/plugins/current/ingest-attachment.html
What if my output of beat is not ES cluster? in that scenario what would be the best way to read JSON?
No, I mean ingest node. Filebeat can parse JSON, so if you are not enriching or transforming the data that may work as well. There is however a limited number of outputs supported, which may limit where you can send data.
It would probably be a lot easier if you could show the data and describe exactly what you want to do with it and where you want to send it.
[{
"connector_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",
"group_guids": ["08fba752-6034-4a38-985a-4819b54b136b"],
"id": 6.419229331435815e+18,
"date": "2018-10-01T13:41:05+00:00",
"file": {
"disposition": "Malicious",
"identity": {
"sha256": "ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa"
}
},
"event_type_id": 5.53648143e+08,
"computer": {
"network_addresses": [{
"mac": "x:x:x:x:x",
"ip": "1.1.1.1"
}],
"links": {
"computer": "https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb",
"trajectory": "https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory",
"group": "https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b"
},
"connector_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",
"hostname": "Demo_WannaCry_Ransomware",
"external_ip": "171.190.158.103",
"active": true
},
"timestamp": 1.538401265e+09,
"detection_id": "6419229327140847660",
"timestamp_nanoseconds": 1.66e+08,
"event_type": "Threat Quarantined"
},
{
"event_type_id": 2.16426088e+09,
"computer": {
"external_ip": "171.190.158.103",
"active": true,
"network_addresses": [{
"ip": "2.2.2.2",
"mac": "y:y:y:y"
}],
"links": {
"trajectory": "https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory",
"group": "https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b",
"computer": "https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb"
},
"connector_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",
"hostname": "Demo_WannaCry_Ransomware"
},
"file": {
"disposition": "Malicious",
"identity": {
"sha256": "ed01ebfbc9eb5bbea545af4d01bf5f1071661840480439c6e5babe8e080e41aa"
}
},
"event_type": "Quarantine Failure",
"timestamp_nanoseconds": 1.66e+08,
"detection_id": "6419229327140847659",
"connector_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",
"id": 6.419229331435815e+18,
"group_guids": ["08fba752-6034-4a38-985a-4819b54b136b"],
"error": {
"description": "Cannot delete",
"error_code": 3.221225761e+09
},
"timestamp": 1.538401265e+09,
"date": "2018-10-01T13:41:05+00:00"
},
{
"group_guids": ["08fba752-6034-4a38-985a-4819b54b136b"],
"event_type_id": 1.090519054e+09,
"detection_id": "6419229327140847667",
"connector_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb",
"date": "2018-10-01T13:41:04+00:00",
"event_type": "Threat Detected",
"detection": "W32.File.MalParent",
"computer": {
"hostname": "Demo_WannaCry_Ransomware",
"external_ip": "171.190.158.103",
"user": "",
"active": true,
"network_addresses": [{
"ip": "3.3.3.3",
"mac": "z:z:z:z"
}],
"links": {
"computer": "https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb",
"trajectory": "https://api.amp.cisco.com/v1/computers/15f51e37-9b50-4b49-834c-0accdff9f5eb/trajectory",
"group": "https://api.amp.cisco.com/v1/groups/08fba752-6034-4a38-985a-4819b54b136b"
},
"connector_guid": "15f51e37-9b50-4b49-834c-0accdff9f5eb"
},
"id": 6.419229327140848e+18,
"timestamp": 1.538401264e+09,
"timestamp_nanoseconds": 7.3e+08,
"file": {
"disposition": "Malicious",
"file_name": "tasksche.exe",
"file_path": "\\?\C:\ProgramData\qzkbplcgew884\tasksche.exe",
"identity": {
"sha256": "ed01ebfbc9eb5bbea545af4d01bf5f10716618404a80439c6e5babe8e080e41aa",
"sha1": "5ff465afaabcbf0150d1a3ab2c2e74f3a4426467",
"md5": "84c82835a5d21bbcf75a61706d8ab549"
}
}
}]
above mentioned is my json which i need to read. I tried option "json.keys_under_root: true", which works well if the json is single line object. but that is not working on above mentioned json.
That is a list of JSON objects, which you will need to break into separate events. For that you will need more advanced processing than Filebeat is capable of. You may need to use Logstash here as I am not sure ingest node pipelines support splitting an event into multiple ones.
if i break each list item in a single line json objects, Should that work? as i tried that but no luck
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.