# Logstash parsing two days behind

**URL:** <https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879>\
**Category:** Logstash\
**Created:** [October 8, 2015, 8:24pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879 "2015-10-08T20:24:00Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [October 8, 2015, 8:24pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/1 "2015-10-08T20:24:00Z")

</div>

Hello, I'm forwarding logs from a remote server to my docker container containing the elk stack and after looking at the logs, it seems they are two days behind. Any reason why this is? Is there a way to fix this?  
Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 8, 2015, 8:37pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/2 "2015-10-08T20:37:52Z")

</div>

You're not giving us enough details for a useful answer. What's the origin of the logs, on-disk files? Are those logs actually being updated? Is Logstash monitoring the right files? What's in the sincedb files? How about the Logstash and Elasticsearch logs? I'd also make sure that the timestamp of the logs is correct. With an incorrect date filter you could have fresh logs being inserted in a two-day old index.

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [October 8, 2015, 8:51pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/3 "2015-10-08T20:51:55Z")

</div>

the log file is one that is constantly getting updated. Logstash has to be pointing to these since they are showing up just a couple days behind. %{HTTPDATE:date} is the filter i am using in logstash. There are no grok parse failures. Elasticsearch is connected and running fine.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 9, 2015, 5:41am UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/4 "2015-10-09T05:41:38Z")

</div>

I'd compare Logstash's current position according to sincedb against the actual size and inocde numbers of the files.

> %{HTTPDATE:date} is the filter i am using in logstash.

Okay, but what does the date _filter_ look like? That's what determines what ends up in `@timestamp`.

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [October 9, 2015, 1:41pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/5 "2015-10-09T13:41:37Z")

</div>

@timestamp is todays date. The date within the logs is two days prior. When i go onto the remote server and check the logs the date matches what is in the @timestamp.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 9, 2015, 2:40pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/6 "2015-10-09T14:40:10Z")

</div>

Okay. And what if you compare the sincedb files with the log files being read? Is Logstash behind there too? Is it catching up in any way?

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [October 9, 2015, 3:22pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/7 "2015-10-09T15:22:29Z")

</div>

which sincedb files?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 9, 2015, 3:24pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/8 "2015-10-09T15:24:04Z")

</div>

The files where Logstash records the current position in each input file. See the [file input documentation](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-file.html) for details.

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [October 9, 2015, 3:26pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/9 "2015-10-09T15:26:10Z")

</div>

im using the lumberjack filter not file though.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 9, 2015, 3:28pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/10 "2015-10-09T15:28:13Z")

</div>

But you use the file input on the remote server where you have the HTTP logs, right?

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [October 9, 2015, 3:33pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/11 "2015-10-09T15:33:18Z")

</div>

"network": {  
"servers": ["IP"],  
"timeout": 15,  
"ssl ca": "/etc/pki/tls/certs/logstash-forwarder.crt"

},

"files": [  
{  
"paths": [  
"/var/log/logfile.log"  
],  
"fields": { "type": "logs1" }  
}  
]  
}  
this is how my forwarder looks on the remote server.. i dont have anything else running on here for the elk stack.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 9, 2015, 3:34pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/12 "2015-10-09T15:34:40Z")

</div>

Oh, right, you use LSF. Well, it has its own state file named .logstash-forwarder (IIRC) that works more or less the same as sincedb (maybe even the same format).

---

<div class="post-metadata">

**Author:** ![kmroz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kmroz/32/62741_2.png) [@kmroz](https://discuss.elastic.co/u/kmroz)\
**Post date:** [October 9, 2015, 3:36pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/13 "2015-10-09T15:36:53Z")

</div>

do you recommend switching to something different than the forwarder?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 9, 2015, 3:41pm UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/14 "2015-10-09T15:41:45Z")

</div>

I'd debug this for a big longer, but you should use whatever works for you. LSF is being deprecated in favor of Filebeat. Log Courier is another option (a fork of LSF).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:26am UTC](https://discuss.elastic.co/t/logstash-parsing-two-days-behind/31879/15 "2017-07-06T05:26:59Z")

</div>


