Strange behavior with ELK setup

Elastic Stack Logstash
1

Here is the setup for my ELK setup. I have about 3 data shippers running on different boxes which ships the log files to central logstash server. The logstash server parses these events and outputs it to Kibana.
I have been trying so hard and not able to get to the bottom of the problem. With this whole setup I see dateParse failures in logstash config. But if I change the logstash output to files instead of elastic server I do not see dateParse failures. I can also confirm that these parsing failures are coming from multliline events in my log files (the ones with error stack trace). The surprising thing is if I process these multiline files individually, it works fine. Not sure how I can resolve this?
I have also increased my multiline interval. It has brought down error count but not resolved all the errors. I am falling short of ideas now.

input {
beats {
port => "5043"
codec => multiline {
pattern => "^\s*[0-9]{4}[-][0-9]{1,2}[-][0-9]{1,2}"
what => "previous"
negate => true
auto_flush_interval => 120
}
}
}

2

Just a heads up, we changed the name from ELK to Elastic Stack, cause otherwise beats gets left out and feels lonely :wink:

Are you using beats as the shippers?

3

Yes I am using filebeat for data shipping.

4

You should use multiline on beats, it'll make things much easier.

5

Thanks! I will check how to use that on beats and would see if it resolves my problem.
Another interesting thing I noted today is, I just started filebeat on 2 servers out of 3 and there were no errors. When all the logs for 2 servers were processed I started filebeat on third server.
Presently there are no errors in my logstash.
My conclusion:- Looks like if lot of data at one go is send to logstash pipeline it messes up. If data is send gradually logstash works.

6

Looks like if lot of data at one go is send to logstash pipeline it messes up. If data is send gradually logstash works.

This might be true if you're using the multiline codec with the beats input. Otherwise no.

7

I am not using multiline in filebeat so not sure why that magic is happening :slight_smile:
For now I am not seeing any errors so I am happy.
Thanks for your reply!

8

I am not using multiline in filebeat so not sure why that magic is happening

I was referring to Logstash's multiline codec.

9

Yes I am using Logstash's multiline input so that means it cannot process lot of data at one go.
When the filebeat on each server is not started simultaneously it works, I mean there is lot of gap in starting filebeat on each server.

Thanks!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.