' for nil:NilClass>, :backtrace=>["(eval):274:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):272:in `block in initialize'", "(eval):291:in `block in initialize'", "org/jruby/RubyArray.java:1734:in `each'", "(eval):286:in `block in initialize'", "(eval):226:in `block in filter\_func'", "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:455:in `filter_batch'", "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:434:in `worker\_loop'", "/home/avk03/JarAndZip/jar-file/config/logstash-6.1.1/logstash-core/lib/logstash/pipeline.rb:393:in `block in start\_workers'"]}
Please Help me :((
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [April 24, 2018, 10:09am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/6 "2018-04-24T10:09:16Z")
`[@metadata][age]`, not `[@timestamp][age]`.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 10:14am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/7 "2018-04-24T10:14:55Z")
Hi thanks for the reply
i changed my configuration to
> age {}
> if [@metadata][age] \> 86400 {
> drop {}
> }
but still it is reading data from 01-22-2018 , it supposed to drop that data.
What's wrong here
---
### Author:  [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)
#### Post date: [April 24, 2018, 11:02am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/8 "2018-04-24T11:02:13Z")
Please show an example of such a document. You can copy/paste from Kibana's JSON tab or use a `stdout { codec => rubydebug }` output.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 11:06am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/9 "2018-04-24T11:06:07Z")
Hi
bellow is the sample document , i am trying to read elb access log data
> {
> "method" =\> "GET",
> "token" =\> "HFboNZI168vuLmSL1521536284629",
> "name" =\> "elb\_access\_log",
> "@timestamp" =\> 2018-01-22T10:50:10.059Z,
> "timestamp" =\> "2018-01-22T10:50:10.059992Z",
> "requestProcessingTime" =\> 4.0e-05,
> "backendProcessingTime" =\> 0.001899,
> "responseProcessingTime" =\> 2.6e-05,
> "httpversion" =\> "1.1",
> "@version" =\> "1",
> "request" =\> "[http://xxxxx.com:80/](http://xxxxx.com:80/)",
> "serverIp" =\> "x.x.x.x",
> "clientIp" =\> "x.x.x.x.",
> "backend\_status\_code" =\> 200,
> "sentBytes" =\> 4438,
> "clientPort" =\> "30818",
> "serverPort" =\> "80",
> "abc" =\> "abc",
> "agent" =\> ""Mozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36"",
> "elb\_status\_code" =\> 200,
> "type" =\> "elb\_access",
> "recivedBytes" =\> 0,
> "geoip" =\> {
> "region\_name" =\> "Maharashtra",
> "continent\_code" =\> "AS",
> "country\_code2" =\> "IN",
> "region\_code" =\> "MH",
> "longitude" =\> 72.8258,
> "city\_name" =\> "Mumbai",
> "timezone" =\> "Asia/Kolkata",
> "location" =\> {
> "lat" =\> 18.975,
> "lon" =\> 72.8258
> },
> "latitude" =\> 18.975,
> "country\_name" =\> "India",
> "country\_code3" =\> "IN",
> "ip" =\> "13.126.167.102"
> },
> "useragent" =\> {
> "os" =\> "Linux",
> "major" =\> "36",
> "build" =\> "",
> "name" =\> "Chrome",
> "patch" =\> "1985",
> "os\_name" =\> "Linux",
> "device" =\> "Other",
> "minor" =\> "0"
> },
> "message" =\> "2018-01-22T10:50:10.059992Z elb\_access\_log x.x.x.x:30818 x.x.x.x:80 0.00004 0.001899 0.000026 200 200 0 4438 "GET [http://xxxxxx.com:80/](http://xxxxxx.com:80/) HTTP/1.1" "Mozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36" - -\n"
> }
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 11:06am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/10 "2018-04-24T11:06:08Z")
`@timestamp` is by default set to the time the event is processed, so unless you are using a `date` filter to set it based on data in your logs prior to the `age` filter, the age is always going to be less than 86400 seconds.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 11:09am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/11 "2018-04-24T11:09:01Z")
@Christian_Dahlqvist
i am parsing the log's time using date filter plugin as bellow
> date {
> match =\> ["timestamp", "ISO8601"]
> }
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 11:09am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/12 "2018-04-24T11:09:56Z")
And you are doing that before you use the `age` filter?
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 11:10am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/13 "2018-04-24T11:10:58Z")
yes , first i'am using `age` filter then `date` filter
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 11:11am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/14 "2018-04-24T11:11:20Z")
You have to do it the other way around. At the point where you are currently applying the `age` filter, the `@timestamp` field is populated with the default value, which is the current time.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 11:15am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/15 "2018-04-24T11:15:40Z")
you mean first i have to parse log time with `date` filter then i have to use `age` filter?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 11:17am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/16 "2018-04-24T11:17:03Z")
Yes.
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 11:17am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/17 "2018-04-24T11:17:40Z")
ok thanks for the reply.... i will try that ,and get back to you please wait
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 11:25am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/18 "2018-04-24T11:25:38Z")
@Christian_Dahlqvist
now it leads to different issue, getting grockparserFailure exception
> {
> "@timestamp" =\> 2018-04-24T11:20:39.360Z,
> "type" =\> "elb\_access",
> "message" =\> "2018-01-22T14:39:14.878876Z elb\_access\_log x.x.x.x:35004 - -1 -1 -1 504 0 0 0 "GET [http://xxxxxxxx:80/index](http://xxxxxxxx:80/index) HTTP/1.1" "Mozilla/5.0 (X11; Linux x86\_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/36.0.1985.67 Safari/537.36" - -\n",
> "@version" =\> "1",
> "tags" =\> [
> [0] "\_grokparsefailure",
> [1] "\_geoip\_lookup\_failure"
> ],
> "token" =\> "HFboNZI168vuLmSL1521536284629"
> }
@timestamp having today's date but in message the time is of January month... ☹
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [April 24, 2018, 11:28am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/19 "2018-04-24T11:28:49Z")
I have no idea what you have changed to get that behaviour. I would have moved the `age` filter to just after the `date` filter. Is that what you did? Can you show your complete config?
---
### Author:  [@RamyaGowda](https://discuss.elastic.co/u/RamyaGowda)
#### Post date: [April 24, 2018, 11:31am UTC](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221/20 "2018-04-24T11:31:19Z")
yeah i did the same thing
> input {
> s3{
> access\_key\_id =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*"
> secret\_access\_key =\> "\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*\*"
> region =\> "us-west-1"
> bucket =\> "abc-logs"
> codec =\> "plain"
> type =\> "access\_logs"
> }
>
> filter {
> if [type] == "elb\_access"
> {
> grok
> {
> match =\> {
> "message" =\> '%{TIMESTAMP\_ISO8601:timestamp} %{DATA:name} %{IPORHOST:clientIp}:%{POSINT:clientPort} %{IPORHOST:serverIp}:%{POSINT:serverPort} %{NUMBER:requestProcessingTime} %{NUMBER:backendProcessingTime} %{NUMBER:responseProcessingTime} %{NUMBER:elb\_status\_code} %{NUMBER:backend\_status\_code} %{NUMBER:recivedBytes} %{NUMBER:sentBytes} "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{QS:agent}'
> }
> }
>
> mutate {
> convert =\> { "recivedBytes" =\> "integer"}
> convert =\> { "backend\_status\_code" =\> "integer"}
> convert =\> { "elb\_status\_code" =\> "integer"}
> convert =\> { "sentBytes" =\> "integer"}
> convert =\> { "requestProcessingTime" =\> "float"}
> convert =\> { "responseProcessingTime" =\> "float"}
> convert =\> { "backendProcessingTime" =\> "float"}
> add\_field =\> { "token" =\> "HFboNZI168vuLmSL1521536284629" }
> }
> geoip {
> source =\> "clientIp"
> }
>
> useragent {
> source =\> "agent"
> target =\> "useragent"
> }
> date {
> match =\> ["timestamp", "ISO8601"]
> locale =\> en
> }
> age {}
> if [@metadata][age] \> 86400 {
> drop {}
> }
> }
> }
> {
> if "\_grokparsefailure" not in [tags]
> {
> elasticsearch {
> hosts =\> "localhost:9200"
> index =\> "abc"
> }
> }
> stdout { codec =\> rubydebug }
> }
My doubts is not about the grok parser , it's about why @timestamp and time present in event both are different??
[Next page](https://discuss.elastic.co/t/skip-reading-historical-data-in-logstash-while-parsing-logs/129221.md?page=2)