One line JSON parsing into separate events/logs

Elastic Stack Beats
1

Hi,

I am trying to ingest some logs via filebeats, the logs get to logstash but only one event is created, the logs are all on one line as per the example below, I have trimmed it down.

{"requests": [{"originId": 283019416, "originType": "Networks", "internalIp": "196.111.111.111", "externalIp": "196.111.111.111", "categories": ["Malware"], "tags": , "destination": "somedomain.com", "originLabel": "primary int", "actionTaken": "BLOCKED", "datetime": "2019-07-16T15:21:49.000Z"}, {"originId": 283019416, "originType": "Networks", "internalIp": "196.111.111.111", "externalIp": "196.111.111.111", "categories": ["Malware"], "tags": , "destination": "somedomain.com", "originLabel": "primary in", "actionTaken": "BLOCKED", "datetime": "2019-07-16T14:18:14.000Z"}]}

An example of my filebeat config is as below

  • type: log
    paths:
    • /opt/logcollection/*.json
      json.keys_under_root: false
      json.add_error_key: true
      tags: ["dns"]

my logstash conf is:
input {
beats {
port => "5044"
tags => [ "beat" ]
codec => "json"
}
}
I have also tried json_lines and the json filter. I have validated the JSON i just cannot seem to get filebeats or logstash to separate the events. Are filebeats/logstash the right place to do this or should I be using the python request to parse the data and create the events on separate lines?

I have gone through a number of threads and the elastic guides, tried a lot of permutations of the settings and I do not seem to be getting anyway and hoping if someone can point me in the right direction.

Thanks
Phil

2

can you please format config and sample events using </>

just to be on the same page. you expect event to be created for each request out of set of requests, right?

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.