No real time logs


#1

So, we have around a huge number of servers approx. 70, whose logs are being forwarded by logstash-forwarder and stored in Elasticsearch. The problem we are facing is the logs are out of order. We are not getting any real time logs as such. Always a delay of around 2-3 hours?
What is the best approach to get the near real time logs?


(Magnus Bäck) #2

With a well-functioning stack you should be able to have logs available within a few seconds. Have you tried to debug where the delay is? To start with, is logstash-forwarder keeping up with the log files? Its registry file containing the current file position can help you with that.


#3

We are using logstash-forwarder on the clients which is forwarding the logs to the server, and yes we do get logs within seconds but not the real time. For example:
" January 18th 2016, 16:35:33.365 message:2016-01-14 22:08:21,543 INFO : com.nostratech.xxx.util.HttpRequest:83 - receive response 200 from http://services-xyz/540259502 "
Even at this point , getting old indices? And how to set the current file position and where?
Thanks for the reply.


(Magnus Bäck) #4

we do get logs within seconds but not the real time

How do you mean? Your example indicates a rather big difference between the time the message was logged and the time Logstash got it (I'm guessing since the meaning of the fields isn't clear).

Even at this point , getting old indices?

It seems logstash-forwarder is reading old log files. Again, without details it's impossible to debug.

And how to set the current file position and where?

I think logstash-forwarder's registry file is named .logstash-forwarder and it's stored in the directory where the program was started.


#5

Yes you are right, but I meant we do get logs but old logs.
Trying to debug and will come back to you with details. Thanks.


#6

Hi Magnus,

I did check the .logstash-forwarder file on some of the servers and it did have old log files.
Logstash is indeed reading old log files. How can I prevent that?


(Magnus Bäck) #7

I don't recall whether LSF starts reading from the beginning or the end of encountered files, and whether the behavior is configurable or not. I do know that Filebeat allows you to configure that, and since LSF has been abandoned you should look into migrating to Filebeat anyway.


#8

I just have one doubt, is it possible that the delay is due to large number of logs messages being sent to the log server all at once.


(Magnus Bäck) #9

Yes, that's a possibility.


#10

Thanks Magnus for the prompt replies.
One more thing, I have started filebeat on some of the servers. How do I make sure that it reads the current log files only ?


(system) #11