No real time logs

So, we have around a huge number of servers approx. 70, whose logs are being forwarded by logstash-forwarder and stored in Elasticsearch. The problem we are facing is the logs are out of order. We are not getting any real time logs as such. Always a delay of around 2-3 hours?
What is the best approach to get the near real time logs?

With a well-functioning stack you should be able to have logs available within a few seconds. Have you tried to debug where the delay is? To start with, is logstash-forwarder keeping up with the log files? Its registry file containing the current file position can help you with that.

We are using logstash-forwarder on the clients which is forwarding the logs to the server, and yes we do get logs within seconds but not the real time. For example:
" January 18th 2016, 16:35:33.365 message:2016-01-14 22:08:21,543 INFO : - receive response 200 from http://services-xyz/540259502 "
Even at this point , getting old indices? And how to set the current file position and where?
Thanks for the reply.

we do get logs within seconds but not the real time

How do you mean? Your example indicates a rather big difference between the time the message was logged and the time Logstash got it (I'm guessing since the meaning of the fields isn't clear).

Even at this point , getting old indices?

It seems logstash-forwarder is reading old log files. Again, without details it's impossible to debug.

And how to set the current file position and where?

I think logstash-forwarder's registry file is named .logstash-forwarder and it's stored in the directory where the program was started.

Yes you are right, but I meant we do get logs but old logs.
Trying to debug and will come back to you with details. Thanks.

Hi Magnus,

I did check the .logstash-forwarder file on some of the servers and it did have old log files.
Logstash is indeed reading old log files. How can I prevent that?

I don't recall whether LSF starts reading from the beginning or the end of encountered files, and whether the behavior is configurable or not. I do know that Filebeat allows you to configure that, and since LSF has been abandoned you should look into migrating to Filebeat anyway.

I just have one doubt, is it possible that the delay is due to large number of logs messages being sent to the log server all at once.

Yes, that's a possibility.

Thanks Magnus for the prompt replies.
One more thing, I have started filebeat on some of the servers. How do I make sure that it reads the current log files only ?