# How should I configure the date filter?

**URL:** <https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985>\
**Category:** Logstash\
**Created:** [October 1, 2016, 10:18am UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985 "2016-10-01T10:18:43Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![espogian](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@espogian](https://discuss.elastic.co/u/espogian)\
**Post date:** [October 1, 2016, 10:18am UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/1 "2016-10-01T10:18:43Z")

</div>

Hi there. I have a .csv file in which my @timestamp field is formatted like this:

```
0043838407D89D6773491A20160918215104.8191+020000

```

How should I configure Logstash to make it ignore the first part and correctly represent the timestamp?

My configuration file is the following:

```
input {
  file {
    path => "path/to/my/file.csv"
    type => "test"
    start_position => "beginning"
  }
}

filter {
  csv {
    columns => [
	    "@timestamp",
# ... a lot of other columns ...
    ]
    separator => ","
  }
  date {
    match => ["@timestamp" , "???"]
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
  }
  stdout { codec => rubydebug }
}

```

Thanks a lot 🙂

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [October 1, 2016, 10:19am UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/2 "2016-10-01T10:19:25Z")

</div>

Where is the actual date in that?

---

<div class="post-metadata">

**Author:** ![espogian](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@espogian](https://discuss.elastic.co/u/espogian)\
**Post date:** [October 1, 2016, 10:20am UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/3 "2016-10-01T10:20:09Z")

</div>

```
20160918215104.8191+020000

```

You see, it's like YYYYMMddHHmmss.ssssZ but I'm unsure it would work and I don't know how to ignore the initial text.  
Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2016, 3:37pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/4 "2016-10-01T15:37:28Z")

</div>

Use a grok filter to extract the pieces you're interested in.

---

<div class="post-metadata">

**Author:** ![espogian](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@espogian](https://discuss.elastic.co/u/espogian)\
**Post date:** [October 1, 2016, 4:19pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/5 "2016-10-01T16:19:07Z")

</div>

You mean using grok instead of csv? Or something else?

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2016, 5:26pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/6 "2016-10-01T17:26:37Z")

</div>

By all means keep using csv, but use a grok filter to extract the interesting parts of the first field. Something like

```nohighlight
grok {
  match => {
    "@timestamp" => ".*(?<@timestamp>\d{14}\.\d+[+-]\d{4})\d\d$"
  }
}

```

would work. If the number of characters that comes before the timestamp is fixed it's possible to write a more exact pattern that's faster.

---

<div class="post-metadata">

**Author:** ![espogian](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@espogian](https://discuss.elastic.co/u/espogian)\
**Post date:** [October 1, 2016, 6:52pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/7 "2016-10-01T18:52:45Z")

</div>

Thanks for the kind reply, I'm starting to understand some nice features 🙂

I have removed the date filter and added your grok filter, loading the following configuration file:

```
input {
  file {
    path => "path/to/my/file.csv"
    type => "test"
    start_position => "beginning"
  }
}

filter {
  csv {
    columns => [
	    "@timestamp",
# ... a lot of other columns ...
    ]
    separator => ","
  }
  grok {
    match => {
      "@timestamp" => ".*(?<@timestamp>\d+\.\d{4}[+-]\d{4})\d\d$"
    }
  }
}

output {
  elasticsearch {
    hosts => ["elasticsearch:9200"]
  }
  stdout { codec => rubydebug }
}

```

But unfortunately, now Logstash is returning me this error:

```
Settings: Default pipeline workers: 2
Pipeline aborted due to error {:exception=>"RegexpError", :backtrace=>["org/jruby/RubyRegexp.java:1434:in `initialize'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/jls-grok-0.11.3/lib/grok-pure.rb:127:in `compile'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:264:in `register'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:259:in `register'", "org/jruby/RubyHash.java:1342:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-grok-2.0.5/lib/logstash/filters/grok.rb:255:in `register'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in `start_workers'", "org/jruby/RubyArray.java:1613:in `each'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:182:in `start_workers'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/pipeline.rb:136:in `run'", "/opt/logstash/vendor/bundle/jruby/1.9/gems/logstash-core-2.4.0-java/lib/logstash/agent.rb:491:in `start_pipeline'"], :level=>:error}
stopping pipeline {:id=>"main"}

```

Can't understand what's going on 😔

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2016, 7:40pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/8 "2016-10-01T19:40:52Z")

</div>

You still need the date filter, and when the `@timestamp` field contains something it can parse it'll actually work. But it seems the regexp library doesn't want to capture into a field with @ in it. So let's call it plain `timestamp` instead. This works:

```nohighlight
filter {
  grok {
    match => {
      "@timestamp" => ".*(?<timestamp>\d{14}\.\d{4}[+-]\d{4})\d\d$"
    }
  }
  date {
    match => ["timestamp", "YYYYMMddHHmmss.SSSSZ"]
    remove_field => ["timestamp"]
  }
}

```

---

<div class="post-metadata">

**Author:** ![espogian](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@espogian](https://discuss.elastic.co/u/espogian)\
**Post date:** [October 1, 2016, 8:20pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/9 "2016-10-01T20:20:10Z")

</div>

Yes now the pipeline is working but there is still something wrong with the date:

`Failed parsing date from field {:field=>"timestamp", :value=>"0043838407D89D6773491A20160918215104.8191+020000", :exception=>"Invalid format: \"0043838407D89D6773491A20160918215104.8191+...\" is malformed at \"D89D6773491A20160918215104.8191+...\"", :config_parsers=>"YYYYMMddHHmmss.SSSSZ", :config_locale=>"default=en", :level=>:warn}`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 1, 2016, 8:23pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/10 "2016-10-01T20:23:06Z")

</div>

It works for me so you're doing something differently:

```nohighlight
$ cat test.config
input { stdin {} }
output { stdout { codec => rubydebug } }
filter {
  grok {
    match => {
      "message" => ".*(?<timestamp>\d{14}\.\d{4}[+-]\d{4})\d\d$"
    }
  }
  date {
    match => ["timestamp", "YYYYMMddHHmmss.SSSSZ"]
    remove_field => ["timestamp"]
  }
}
$ echo '0043838407D89D6773491A20160918215104.8191+020000' | logstash -f test.config
Settings: Default pipeline workers: 8
Pipeline main started
{
       "message" => "0043838407D89D6773491A20160918215104.8191+020000",
      "@version" => "1",
    "@timestamp" => "2016-09-18T19:51:04.819Z",
          "host" => "bertie"
}
Pipeline main has been shutdown
stopping pipeline {:id=>"main"}

```

---

<div class="post-metadata">

**Author:** ![espogian](https://avatars.discourse-cdn.com/v4/letter/e/4491bb/32.png) [@espogian](https://discuss.elastic.co/u/espogian)\
**Post date:** [October 1, 2016, 8:38pm UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/11 "2016-10-01T20:38:08Z")

</div>

Got it! I forgot to change the field name inside the grok filter.  
Thanks for your kind support, this topic greatly improved my (little) knowledge 🙂

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:36am UTC](https://discuss.elastic.co/t/how-should-i-configure-the-date-filter/61985/12 "2017-07-06T04:36:12Z")

</div>