Elapsed filter inquiry

Hello, I created an elapsed filter, but I can't find the time elapsed in the stdout neither elasticsearch/kibana, I only find those lines in stdout:

logstash_1       | [2017-06-20T15:12:48,787][INFO ][logstash.filters.elapsed ] Elapsed, 'start event' received {:start_tag=>"taskStarted", :unique_id_field=>"task_id"}
logstash_1       | [2017-06-20T15:12:48,848][INFO ][logstash.filters.elapsed ] Elapsed, 'end event' received {:end_tag=>"taskTerminated", :unique_id_field=>"task_id"}

I tried it also with new_event_on_match => true but it remains the same, I am sure that my start and end patterns are extracted correctly in grok.

Does this mean elapsed filter couldn't match the start and end or something else?

any ideas?

Are you converting the date field in the [] e.g. [2017-06-20T15:12:48,848] to the event timestamp using the date filter?

Hello @guyboertje, sorry for the late response, I was on a vacation.
I tried adding date filter but the output remains the same.

here's a snap of my log file:

2017-01-01 07:53:44 [utils.py] WARNING: enable_proxy must have atleast one http
2017-01-01 07:53:45 [provider.py] DEBUG: Using access key found in environment variable.
2017-01-01 07:53:50 [engine.py] INFO: Spider opened
2017-01-01 07:54:01 [logstats.py] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-01 07:55:44 [monitor_utils.py] INFO: Getting the latest iteration for merchant ariika

I am trying to get the duration between the first and last lines.

here's my logstash configuration:

input {
    tcp {
        port => 5000
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous

filter {

    			match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} WARNING: enable_proxy must %{GREEDYDATA:task_id}"]
  				add_tag => [ "taskStarted" ]

  				match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} INFO: Getting the latest iteration for %{GREEDYDATA:task_id}"]
  				add_tag => [ "taskTerminated"]

				match => [ "message", "%{DATE_EU:timestamp}" ]
		    	match => [ "timestamp", "yy-MM-dd" ]
		   		target => "@timestamp"

    			start_tag => "taskStarted"
    			end_tag => "taskTerminated"
    			unique_id_field => "task_id"
    			new_event_on_match => true

output {
	if "_grokparsefailure" not in [tags]{
		stdout {
		codec => rubydebug

You are not converting the full date time text into a timestamp - the event should have a_pateparsefailure tag.
Also, the date filter uses @timestamp as a target by default so no need to specify the target.

Use the inbuilt ISO8601 date filter support.

		    	match => [ "timestamp", "ISO8601" ]

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.