# Skip reading historical data in logstash while parsing logs

**URL:** <https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221>\
**Category:** Logstash\
**Created:** [April 24, 2018, 6:01am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221 "2018-04-24T06:01:49Z")\
**Posts on this page:** 6\
**Page:** 2

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 24, 2018, 11:44am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/21 "2018-04-24T11:44:11Z")

</div>

As the grok parsing failed, `@timestamp` could not be updated based on the data in the log entry, which is why you see the default value (the current processing time).

I would also recommend moving the `date` and `age` filters to just after the `grok` filter. That way you can drop the events before going through the `deep` and `useragent` processing.

---

<div class="post-metadata">

**Author:** ![RamyaGowda](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramyagowda/32/27440_2.png) [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)\
**Post date:** [April 24, 2018, 11:50am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/22 "2018-04-24T11:50:20Z")

</div>

ok i got it, but why it is reading 3 months older data ??

> {  
> "token" =\> "HFboNZI168vuLmSL1521536284629",  
> "@version" =\> "1",  
> "tags" =\> [  
> [0] "\_grokparsefailure",  
> [1] "\_geoip\_lookup\_failure"  
> ],  
> "type" =\> "elb\_access",  
> "message" =\> "2018-01-22T14:22:55.661636Z awseb-e-m-AWSEBLoa-1EJARVYCNV2RH x.x.x.x:57784 - -1 -1 -1 504 0 0 0 "GET [https://xxxxxxx:443/](https://xxxxxxx:443/) HTTP/1.1" "Mozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/61.0.3163.100 Safari/537.36" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2\n",  
> "@timestamp" =\> 2018-04-24T11:43:27.472Z  
> }

Please see the message , this event is of January month.

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 24, 2018, 11:54am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/23 "2018-04-24T11:54:55Z")

</div>

That is probably because the `s3` input plugin reads all files in that bucket, which probably contains older data. I have not used the `s3` plugin, but it looks like you might be able to exclude files/data to be read using the [exclude\_pattern](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-s3.html#plugins-inputs-s3-exclude_pattern) directive.

---

<div class="post-metadata">

**Author:** ![RamyaGowda](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramyagowda/32/27440_2.png) [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)\
**Post date:** [April 24, 2018, 11:56am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/24 "2018-04-24T11:56:08Z")

</div>

ok thank you .... i will try it

---

<div class="post-metadata">

**Author:** ![RamyaGowda](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ramyagowda/32/27440_2.png) [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)\
**Post date:** [April 25, 2018, 9:58am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/25 "2018-04-25T09:58:05Z")

</div>

@Christian_Dahlqvist

Thanks for your help and time, i solved the issue's and now `age` filter is working great 😃

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 23, 2018, 9:58am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/26 "2018-05-23T09:58:16Z")

</div>

Closed on May 23, 2018, 9:58am UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221.md?page=1)