Logstash multiline and clone

[adrianfusco (Adrian)]

I am sending some information from filebeat using one kind of multiline pattern.

At some point, I realized I need to use another multiline pattern based on the log_source.

My idea was clone the beat and if the log_source match the one I need and the type is clone, apply another multiline pattern on this one to send the information into different documents and not a single one into another index.

The thing is that I think I should apply it in the filtering side for being able to do it, but this plugin has been deprecated: Multiline filter plugin | Logstash Reference [8.5] | Elastic

Something like this:

input {
    beats {
        port => 5044
    }
}

filter {
    if [log_source] == "logs_for_filter" {
        clone  {
            clones => ["cloned_logs_for_filter"]
        }
    }
    
    if [type] == "cloned_logs_for_filter" {
        multiline {
            pattern => "(^<case)"
            negate => true
            what => previous
        }
    }
    if [log_source] == "logs_for_filter" and "cloned_logs_for_filter" not in [type] {
       # do things
    }
}
output {
    stdout {
        codec => rubydebug
    }
    if [type] == "cloned_logs_for_filter" {
        elasticsearch {
            hosts => "elasticsearch:9200"
            index => "index_2"
        }
    }
    else {
        elasticsearch {
            hosts => "elasticsearch:9200"
            index => "index_two"
        }
    }
}

Do you think there's another way to do it? Thank you.

[leandrojmp (Leandro Pereira)]

It won't work this way.

You need to configure multiline in the source, in this case, you need to configure it in beats.

The best option would be to use a different log input for each kind of multiline you have.

[adrianfusco (Adrian)]

That's another good solution @leandrojmp thank you very much.

--

At the end I've applied the split plugin

        split {
            field => "message"
            terminator => "
"
        }

And then pass the grok filter in each one of them. At this way it's storing each match in different documents.