Multiline codec in input

[rmalired]

I have a scenario where filebeat send information from access.log as well as application.log. The application log can be multiline.

so in the logstash.conf i have a multiline codec looking for a timestamp as the pattern.

input {

beats {
 type => beats
 port => 5044
     codec => multiline{
      pattern => "(^\<%{MONTHDAY}\s%{MONTH}\s%{YEAR}\s%{HOUR}:%{MINUTE}:%{SECOND}\>)"
      negate => true
      what => "previous"
     }

}
}

but the access.log starts with IP address when I ingest access.log data to logstash it treats all the lines into one message

what is the best solution for this kind of situation

I see couple of pointers

Mixed JSON input and Other Log files with Multiline codec

It's pretty clear that your current multiline pattern won't work for JSON lines that begin with a curly brace. Would changing the pattern to e.g. ^(\d+|\{") help?

or use

filebeat multiline, I don't want to do this approach because I have lot of harvesters on application log files (20 micro services deployed on a single container)

thanks
rakesh

[warkolm (Mark Walkom)]

@rmalired

or use

filebeat multiline

This is the best option.

@rmalired

I don't want to do this approach because I have lot of harvesters on application log files (20 micro services deployed on a single container)

You don't have automation to handle it?

[rmalired]

Mark,

Ended up using multiline filebeat.

Automation is there, My only concern was how much additional cpu and memory will be consumed by filebeat, if I put the multiline logic onto the client machine.