So I currently have about 3-5 log files that have multiline in them which is normal. I usually start parsing them 1 file at a time in separate logstash instances and everything runs great no grokparsefailures. The problem starts occurring when I try to combine these configurations into one big config file then grokparsefailures start showing, up not many just about 5 out of like thousands. I do not understand why?! It works when I only test 1 file but the instant I try to input 3-5 files and filter through them then grokparsefailures, though I know it works when there is just single input. Weird right?! Anyone else facing this? I am starting to think its my problem and not some weird error but every time its a different grokparsefailure, meaning the multiline stops at random points of a trace... I need some help.
Can you post your configuration example. ?
Sorry for lateness is this common config its what I am doing:
filter {
if [message] =~ (remove blank lines code)
if [type] == "jtrace" {
multiline {
pattern => "^\s"
what => "previous"
}
} else if [type] == "server" {
multiline {
pattern => "\d{4}-\d{2}-\d{2}....gets pretty specific"
}
} ... there is more multline code
}
then it goes off into actually using grok. Not sure if this is the intented plan
My guess is that your running in to a small issue related to threading. Did you add any options to the CLI of logstash?
If Multiline is the item which your having issue with then you might try the multiline codec on the file input rather then a filter.
If you depend upon multiple lines in your grok then that could be one issue. I can not advise without seeing the whole thing as it depends on how you set up your if statements, and the actual grok statements.
Nope no other option in the CLI except for selecting the log conf.
Cannot use the multiline codec since it will not read the last line
and my other filter is just
filter --> if type match --> grok parse it --> add some date