# No real time logs **URL:** \ **Category:** Elasticsearch\ **Created:** [January 18, 2016, 9:13am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428 "2016-01-18T09:13:43Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![novice](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@novice](https://discuss.elastic.co/u/novice)\ **Post date:** [January 18, 2016, 9:13am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/1 "2016-01-18T09:13:43Z")
So, we have around a huge number of servers approx. 70, whose logs are being forwarded by logstash-forwarder and stored in Elasticsearch. The problem we are facing is the logs are out of order. We are not getting any real time logs as such. Always a delay of around 2-3 hours? What is the best approach to get the near real time logs? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2016, 9:26am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/2 "2016-01-18T09:26:59Z")
With a well-functioning stack you should be able to have logs available within a few seconds. Have you tried to debug where the delay is? To start with, is logstash-forwarder keeping up with the log files? Its registry file containing the current file position can help you with that. ---
**Author:** ![novice](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@novice](https://discuss.elastic.co/u/novice)\ **Post date:** [January 18, 2016, 9:39am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/3 "2016-01-18T09:39:07Z")
We are using logstash-forwarder on the clients which is forwarding the logs to the server, and yes we do get logs within seconds but not the real time. For example: " January 18th 2016, 16:35:33.365 message:2016-01-14 22:08:21,543 INFO : com.nostratech.xxx.util.HttpRequest:83 - receive response 200 from [http://services-xyz/540259502](http://services-xyz/540259502) " Even at this point , getting old indices? And how to set the current file position and where? Thanks for the reply. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2016, 9:46am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/4 "2016-01-18T09:46:38Z")
> we do get logs within seconds but not the real time How do you mean? Your example indicates a rather big difference between the time the message was logged and the time Logstash got it (I'm guessing since the meaning of the fields isn't clear). > Even at this point , getting old indices? It seems logstash-forwarder is reading old log files. Again, without details it's impossible to debug. > And how to set the current file position and where? I think logstash-forwarder's registry file is named .logstash-forwarder and it's stored in the directory where the program was started. ---
**Author:** ![novice](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@novice](https://discuss.elastic.co/u/novice)\ **Post date:** [January 18, 2016, 9:51am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/5 "2016-01-18T09:51:08Z")
Yes you are right, but I meant we do get logs but old logs. Trying to debug and will come back to you with details. Thanks. ---
**Author:** ![novice](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@novice](https://discuss.elastic.co/u/novice)\ **Post date:** [January 18, 2016, 10:37am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/6 "2016-01-18T10:37:03Z")
Hi Magnus, I did check the .logstash-forwarder file on some of the servers and it did have old log files. Logstash is indeed reading old log files. How can I prevent that? ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2016, 11:28am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/7 "2016-01-18T11:28:15Z")
I don't recall whether LSF starts reading from the beginning or the end of encountered files, and whether the behavior is configurable or not. I do know that Filebeat allows you to configure that, and since LSF has been abandoned you should look into migrating to Filebeat anyway. ---
**Author:** ![novice](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@novice](https://discuss.elastic.co/u/novice)\ **Post date:** [January 18, 2016, 4:47pm UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/8 "2016-01-18T16:47:26Z")
I just have one doubt, is it possible that the delay is due to large number of logs messages being sent to the log server all at once. ---
**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\ **Post date:** [January 18, 2016, 6:13pm UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/9 "2016-01-18T18:13:47Z")
Yes, that's a possibility. ---
**Author:** ![novice](https://avatars.discourse-cdn.com/v4/letter/n/a87d85/32.png) [@novice](https://discuss.elastic.co/u/novice)\ **Post date:** [January 19, 2016, 11:18am UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/10 "2016-01-19T11:18:15Z")
Thanks Magnus for the prompt replies. One more thing, I have started filebeat on some of the servers. How do I make sure that it reads the current log files only ? ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 5, 2017, 11:23pm UTC](https://discuss.elastic.co/t/no-real-time-logs/39428/11 "2017-07-05T23:23:31Z")