Need a clarification regarding filebeat & multiline codec

Elastic Stack Beats
1

Currently I am running a filebeat instance on one client. The output of filebeat is directed to logstash and a FQDN is entered for logstash hosts. This FQDN can resolve to 3 IP's where 3 instances of logstash are running. So in this scenario there is a chance that every event can get sent to every other logstash instance running on 3 different servers randomly depending on FQDN resolution. This is fine as the 3 servers are part of a cluster but I am assuming the problem might arise when we use multi line codec on the input section of logstash config. See below the logstash config:

input {
         beats {
                port => "5033"
                        codec => multiline {
                        negate => "true"
                        pattern => "(\.MAJOR)"
                        what => "previous"
                        }
                }
}

Since I have used multiline codec on the input section, it would work fine as long all events are sent to just one logstash instance. But what would happen in my case where FQDN might resolve randomly to any logstash instance and every other event is sent to every other logstash instances running on 3 different servers? I am assuming that the events in multiline codec might match incorrectly or might even not match at all. Please clarify my doubt. In the meantime I still haven't tried this but I think I might run into this issue when I bring my setup up and running.

2

Indeed, the configuration above will only work if Filebeat connects to a single Logstash server. Multiline support is on the way in for Filebeat and that's what you want to go for.

3

Thanks Magnus. I just happened to come across this link which is a ticket in filebeat to bring in the support for multiline codec. That's great news BTW. Well the ticket says the support is ready and ticket is closed. Do you think I can use the latest nightlies to try this feature out?

4

Yes, in that case the feature should be ready in the nightlies (or if you build from master yourself).

5

The feature will be shipped with filebeat 1.1. If you prefer something as close to 1.1 as possible and more stable, build it from the 1.1 branch, or if you are fine with the nightlies, pick it from here: https://beats-nightlies.s3.amazonaws.com/index.html?prefix=filebeat/

6

Thanks Nicolas :slight_smile: