Logstash Does Not Read In Last Line of Log

Elastic Stack Logstash
1

So recently noticed that my logstash config is not reading in the last line from my log file. I think that it may be my input config fault though I do not see any problems with it right now. What else could it be? Here is my input config:
input {

file {
type => "java"
codec => multiline {
pattern => "^\s"
what => "previous"
}
path => "something"
start_position => "beginning"
sincedb_path => "somewhere"
}
file {
type => "communication"
codec => multiline {
pattern => "cool pattern"
negate => "true"
what => "previous"
}
path => "somwhereelse"
start_position => "beginning"
sincedb_path => "somwhereelse"
}

file {
type => "sometype"
codec => multiline {
pattern => "coolpattern"
negate => "true"
what => "previous"
}
path => "somwhereelse"
start_position => "beginning"
sincedb_path => "somwhereelse"
}

}

After much research fixed it. Had to read a lot of how they personally attacked the problem. But if someone is looking for solution just reply back.

2

Providing your solution would help others!

3

Its just a really simple fix but I could not find it in I guess modern forms/community. Had to dig deep into other chats. So basically you move the multiline command to the filter rather then the codec that is used in the input. You should notice that in the 1.5.0 release that multiline is placed in filter, but I read somewhere that people were using multiline in input as a codec. Bad mistake...
(Problem) Multiline is waiting for the last line but cannot determine the last line is indeed the last line so it has to wait for another log to be read in. Problem real time errors will not show up...slow logs...more bad stuff
(Fix) So now there is an update such that there will be a time out on the last log. Now you will never miss your last log. But you have to make sure that you MOVE multiline into FILTER and not input. I tried leaving the command within input block and problem persists. GL to everyone else learning like me.

4

Apologies @michaellizhou for opening an old thread. But i have config where in i have the multiline as a filter and still the last line of the log is not considered.

input{
stdin {
}
}

filter {

multiline {
pattern => "^%{DATE}"
negate => true
what => previous
}

grok {
match =>
[
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{DATA}Run:%{DATA}%{NUMBER:runNo:int},%{DATA}ContextKey:%{DATA}%{NUMBER:ctx:int},%{DATA}COBDate:%{DATA}(?%{MONTHDAY}-%{MONTH}-%{YEAR}),%{DATA}Step:%{GREEDYDATA:step},%{DATA}Type:%{GREEDYDATA:runType},%{DATA}Status:%{DATA}%{WORD:status},%{DATA}Time:%{DATA}%{NUMBER:timeTaken:int}%{GREEDYDATA}",
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{DATA}Run:%{DATA}%{NUMBER:runNo:int},%{DATA}ContextKey:%{DATA}%{NUMBER:ctx:int},%{DATA}COBDate:%{DATA}(?%{MONTHDAY}-%{MONTH}-%{YEAR}),%{DATA}Step:%{GREEDYDATA:step},%{DATA}Type:%{GREEDYDATA:runType},%{DATA}Status:%{DATA}%{WORD:status}%{GREEDYDATA}",
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{GREEDYDATA}"
]

}

date {
match => ["loggedtime","MM/dd/yyyy HH:mm"]
}

}

output {
stdout {
codec=>rubydebug
}
}

and the sample data is :

06/16/15 11:30-XX-INFO: Run:1,ContextKey: 1, COBDate: 31-May-2015, Step: X, Type: Whole,Status:Running
06/16/15 11:40-XX-INFO: Run:1, ContextKey: 1, COBDate: 31-May-2015, Step: X 1, Type: Individual, Status: Running

Request your help on this

5

Quick question what version of logstash are you use?

6

I am using logstash 1.5.0 , Apologies for missing that.

7

That is just really odd because I know they patched up the last log issue. Can you paste a sample of what you are getting for an output? I quickly threw your config into logstash and I am getting an error.

The error reported is: 
  undefined group option: /(?(?:(?:(?:(?:0?[1-9]|1[0-2]))[\/-](?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[\/-](?:(?>\d\d){1,2}))|(?:(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))[.\/-](?:(?:0?[1-9]|1[0-2]))[.\/-](?:(?>\d\d){1,2})))(?:\s*)(?(?:(?:2[0123]|[01]?[0-9])):(?:(?:[0-5][0-9]))))-(?(?:\b\w+\b)-(?<LOGLEVEL:loggingLevel>([Aa]lert|ALERT|[Tt]race|TRACE|[Dd]ebug|DEBUG|[Nn]otice|NOTICE|[Ii]nfo|INFO|[Ww]arn?(?:ing)?|WARN?(?:ING)?|[Ee]rr?(?:or)?|ERR?(?:OR)?|[Cc]rit?(?:ical)?|CRIT?(?:ICAL)?|[Ff]atal|FATAL|[Ss]evere|SEVERE|EMERG(?:ENCY)?|[Ee]merg(?:ency)?))):(?:.*?)Run:(?:.*?)(?<NUMBER:runNo:int>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))),(?:.*?)ContextKey:(?:.*?)(?<NUMBER:ctx:int>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+)))))),(?:.*?)COBDate:(?:.*?)(?(?:(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]))-(?:\b(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\b)-(?:(?>\d\d){1,2})),(?:.*?)Step:(?<GREEDYDATA:step>.*),(?:.*?)Type:(?<GREEDYDATA:runType>.*),(?:.*?)Status:(?:.*?)(?<WORD:status>\b\w+\b),(?:.*?)Time:(?:.*?)(?<NUMBER:timeTaken:int>(?:(?:(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\.[0-9]+)?)|(?:\.[0-9]+))))))(?:.*)/m

So what ever that means its not liking it when I try on my logstash. Maybe try to upload your config on github or anything else that is easier to copy and paste the code.

8

Thanks a lot for taking your time and trying it out. You Could keep this small GROK pattern, instead of the list of the patterns i had.

input {
stdin {
}
}
grok {
match =>
[
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{GREEDYDATA}"
]

}

output {
stdout {
codec=>rubydebug
}
}

I have specified the sample data in the first request itself. Please do let me know if you need any more information from my side.

9

Same problem btw your missing the filter part in the config you just pasted. I do not know if your config is compiling for you or not but you should check with --configtest and then try to actually run it. I can say its definitely not compiling on my end. Must be something up with your grok pattern. Since it seems like you are not that familiar with grok patterns I would recommend doing this:
Write the pattern yourself! it might be time consuming at first but you learn a lot about regex, but once you figure this out then start using some of the default patterns that is provided by grok. That is at least what I did. What your config gives me

grokissues.png1920×1080 1.73 MB

here is quick start to your date parsing. you know what to do from here

"message" , "(?<date>\d+\/\d+\/\d+s*\d+:\d+)...
10

Thanks again for your time @michaellizhou . One thing, is there is a small error in the configuration which i sent. Its a copy-paste issue. I just wanted to let you know that i am working on Windows platform and below is the configtest result

C:\Users\o558781\Desktop>logstash.bat -f "C:\Users\o558781\Desktop\logstashconf.conf" --configtest
io/console not supported; tty will not be manipulated
Configuration OK

Below is the config file i have used

C:\Users\o558781\Desktop>type logstashconf.conf

input {
stdin {
}
}

filter {
grok {
match =>
[
"message","(?%{DATE}%{SPACE}(?%{HOUR}:%{MINUTE}))-(?%{WORD}-%{LOGLEVEL:loggingLevel}):%{GREEDYDATA}"
]
}
}
output {
stdout {
codec=>rubydebug
}
}

Please do let me know if you need any more information from my side

11

i have the same problem and don't know how to fix it.
my log file has 6 line ,but the stdout only print 5 line of it, after i add a new line to the log file , the stdout print the 6 line.

12

Might be be better to start a new thread, this one is 9 months old :slight_smile:

13

great solution ,
had the same problem , messages were not commimg in on-line

DO NOT USE MULTILINE AS AN INPUT CODEC!!!

14

Multiline filter has been deprecated in favor of multiline-codec But the problem still persist with codec .
Does anybody has any workaround ? Last line of the logs is not getting processed

input {
beats {
port => 5044
codec => multiline {
pattern => "^\s"
negate => "true"
what => "previous"
}
}
}

15

Is there any solution to this with multiline codec? I am using logstash 5.2 and having the same issue. Unable to read the last line.

16

I used the following configuration of the codec multiline and it appears to work correctly. Just set :

auto_flush_interval => 1

3 Likes