Logstash not reading complete Tomcat catalina log files

Hi , I have the following logstash configuration file to read catalina log files:-

input {
file {
type => "tomcat"
path => [ "D:/krushnat/catalina.2016-03-16.log" ]
codec => multiline {
negate => true
pattern => "(^%{MONTH} %{MONTHDAY}, 20%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}) (?:AM|PM))"
what => "previous"
}
}
}
filter {

if [type] == "tomcat" {
    
    grok{
       
         match => [ "message", "%{CATALINA_DATESTAMP:timestamp} %{NOTSPACE:className} %{WORD:methodName}\r\n%{LOGLEVEL: logLevel}: %{GREEDYDATA:message}" ]
         overwrite => [ "message" ]
    }
	date{
		match=>["timestamp","MMM dd, yyyy HH:mm:ss a"]
	}
   
}

}
output {
stdout { codec=>rubydebug }
elasticsearch{
hosts=>"localhost"
index=>"catalina_logs_20160316"
}
}

  1. Every time I check kibana all the records are read except last record in each and every catalina file.
  2. So I add one more record at the end of the catalina file.
  3. In Kibana, now previous unread record is now read and new record which I have added is not read.
    What is this behaviour of Logstash which I am failing to understand ??

Until the multiline codec sees the beginning of a new event it can't possibly know whether the current event is finished. What you can do is set the auto_flush_interval option to indicate how long the codec should wait before assuming that the current event is done and won't receive additional lines. The documentation is unclear on the unit of auto_flush_interval but I think it's seconds. Normally all lines of a multiline message is logged in one swoop (most likely a single write operation) so you can have a quite low number here.

Hi

I have the same problem but I have already use the option auto_flush_interval as above:

tcp {
type => "log4j"
port => 5516
codec => multiline {
pattern => "^%{TIMESTAMP_ISO8601} "
what => "previous"
negate => true
auto_flush_interval => 5
}
}

But nothing, the last string stay in a wait mode until a new one comes or the program.
I have no idea, the code seem correct to me.

Thank you

Which version of Logstash are you using?

I'm using the 2.2.2 version