Add field for source - JSON file via filebeats

Elastic Stack Logstash
1

Hi,

I am receiving a JSON file with Logstash via the filebeats agents from multiple remote systems. hostabc, hostdef, and so on.

  • I have had the problem that the received JSON file is escaped once Logstash has written it. Therefore I added the format after the line codec. I don't know how to do this in a better way and the CPU utilization is quite high for this. If someone knows a better way, please let me know :wink:

  • Apart from this I want to tag the source host, which sent the file. In order to do this I need to add a field to the line. Something like:

hostabc - { json stuff }
hostdef - { json stuff }

Preferably I'd like this as an addional JSON field

{ "source":"hostabc", rest of the json stuff }

Here is how this looks:

input {
        beats {
                port => 5044
        }
}

output {
        file {
                path => "/var/log/suricata/eve-remote.json"
                codec => line { format => "%{message}"}
        }
}

The messages look like this:

{"timestamp":"2016-08-16T14:43:16.908358+0000","flow_id":3753490846 ... }

FluentD could do this. I am not sure if Logstash is that capable...

2

What do you currently get in eve-remote.json? What would you like to get instead?