Running multiple filebeat instances

Elastic Stack Beats
1

Hi,

I am currently using Filebeat to send both NGINX and Syslog towards an ELK stack, which works perfectly. Nice! Now I am trying to add OSSEC to the equation. The OSSEC files are in JSON format, while the other logs are not. This creates a problem at Logstash Input, working with two codec types on the same inbound TCP port.

I did read some comments about running two Filebeat instances. Meaning I could then use different Filebeat configs and therefore two different TCP ports to output towards Logstash. Not so much of a problem for me. The only question to a Linux newbie is how to actually run two instances in daemon mode? I am guessing that would entail copying the provided script and amending it to create a second instance? Or is there another way?

Thanks for your excellent work, this is a great help to us here!

Regards,
Reinier

2

I don't think you need 2 instances. Just configure 2 prospectors. With version 5.0 alpha1(or alpha2) the json can even be parsed by filebeat (if 1 json log per line).

In every prospector one can use additional fields/tags for use with logstash. e.g. (filebeat versions 1.x):

filebeat:
  prospectors:
    - input_type: log
      paths:
         - ...
      fields_under_root: true
      fields:
        json: false
    - input_type: log
      paths:
         - ...
      fields_under_root: true
      fields:
        json: true

Then I can use json filter in logstash:

filter{
    if [json] {
        json {
            source => "message"
        }
    }
}
3

Flawless victory, Steffen :slight_smile: Just finished reconfiguring our servers. Very nice. Off to catch some shut-eye. Thanks a lot for your help!

Cheers,
Reinier