# Elapsed filter inquiry

**URL:** <https://discuss.elastic.co/t/elapsed-filter-inquiry/90118>\
**Category:** Logstash\
**Created:** [June 20, 2017, 3:17pm UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118 "2017-06-20T15:17:21Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [June 20, 2017, 3:17pm UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118/1 "2017-06-20T15:17:21Z")

</div>

Hello, I created an elapsed filter, but I can't find the time elapsed in the stdout neither elasticsearch/kibana, I only find those lines in stdout:

```
logstash_1 | [2017-06-20T15:12:48,787][INFO][logstash.filters.elapsed] Elapsed, 'start event' received {:start_tag=>"taskStarted", :unique_id_field=>"task_id"}
logstash_1 | [2017-06-20T15:12:48,848][INFO][logstash.filters.elapsed] Elapsed, 'end event' received {:end_tag=>"taskTerminated", :unique_id_field=>"task_id"}

```

I tried it also with `new_event_on_match => true` but it remains the same, I am sure that my start and end patterns are extracted correctly in grok.

Does this mean elapsed filter couldn't match the start and end or something else?

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [June 22, 2017, 2:56pm UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118/2 "2017-06-22T14:56:47Z")

</div>

any ideas?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [June 22, 2017, 3:14pm UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118/3 "2017-06-22T15:14:20Z")

</div>

Are you converting the date field in the `[]` e.g. `[2017-06-20T15:12:48,848]` to the event timestamp using the date filter?

---

<div class="post-metadata">

**Author:** ![ibrahimsharaf](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ibrahimsharaf/32/17304_2.png) [@ibrahimsharaf](https://discuss.elastic.co/u/ibrahimsharaf)\
**Post date:** [June 30, 2017, 8:44pm UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118/4 "2017-06-30T20:44:23Z")

</div>

Hello @guyboertje, sorry for the late response, I was on a vacation.  
I tried adding date filter but the output remains the same.

here's a snap of my log file:

```
2017-01-01 07:53:44 [utils.py] WARNING: enable_proxy must have atleast one http
2017-01-01 07:53:45 [provider.py] DEBUG: Using access key found in environment variable.
2017-01-01 07:53:50 [engine.py] INFO: Spider opened
2017-01-01 07:54:01 [logstats.py] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-01 07:55:44 [monitor_utils.py] INFO: Getting the latest iteration for merchant ariika

```

I am trying to get the duration between the first and last lines.

here's my logstash configuration:

```
input {
    tcp {
        port => 5000
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
        }
    }
}

filter {

		grok{
    			match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} WARNING: enable_proxy must %{GREEDYDATA:task_id}"]
  				add_tag => ["taskStarted"]
  		}

  		grok{
  				match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} INFO: Getting the latest iteration for %{GREEDYDATA:task_id}"]
  				add_tag => ["taskTerminated"]
		}

		grok{
				match => ["message", "%{DATE_EU:timestamp}"]
			}
			
		date{
		    	match => ["timestamp", "yy-MM-dd"]
		   		target => "@timestamp"
			}

		elapsed{
    			start_tag => "taskStarted"
    			end_tag => "taskTerminated"
    			unique_id_field => "task_id"
    			new_event_on_match => true
  		}
	}

output {
	if "_grokparsefailure" not in [tags]{
		stdout {
		codec => rubydebug
		}
         } 
 }
```

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [July 3, 2017, 9:34am UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118/5 "2017-07-03T09:34:59Z")

</div>

> [@ibrahimsharaf](#):
>
> ```auto
> date{
> match => ["timestamp", "yy-MM-dd"]
> target => "@timestamp"
> }
> 
> ```

You are not converting the full date time text into a timestamp - the event should have a`_pateparsefailure` tag.  
Also, the date filter [uses `@timestamp` as a target by default](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-target) so no need to specify the target.

Use the [inbuilt ISO8601 date filter support](https://www.elastic.co/guide/en/logstash/current/plugins-filters-date.html#plugins-filters-date-match).

```auto
            date{
		    	match => ["timestamp", "ISO8601"]
			}

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 31, 2017, 9:44am UTC](https://discuss.elastic.co/t/elapsed-filter-inquiry/90118/6 "2017-07-31T09:44:46Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.