Elapsed filter inquiry

Elastic Stack Logstash
1

Hello, I created an elapsed filter, but I can't find the time elapsed in the stdout neither elasticsearch/kibana, I only find those lines in stdout:

logstash_1       | [2017-06-20T15:12:48,787][INFO ][logstash.filters.elapsed ] Elapsed, 'start event' received {:start_tag=>"taskStarted", :unique_id_field=>"task_id"}
logstash_1       | [2017-06-20T15:12:48,848][INFO ][logstash.filters.elapsed ] Elapsed, 'end event' received {:end_tag=>"taskTerminated", :unique_id_field=>"task_id"}

I tried it also with new_event_on_match => true but it remains the same, I am sure that my start and end patterns are extracted correctly in grok.

Does this mean elapsed filter couldn't match the start and end or something else?

2

any ideas?

3

Are you converting the date field in the [] e.g. [2017-06-20T15:12:48,848] to the event timestamp using the date filter?

4

Hello @guyboertje, sorry for the late response, I was on a vacation.
I tried adding date filter but the output remains the same.

here's a snap of my log file:

2017-01-01 07:53:44 [utils.py] WARNING: enable_proxy must have atleast one http
2017-01-01 07:53:45 [provider.py] DEBUG: Using access key found in environment variable.
2017-01-01 07:53:50 [engine.py] INFO: Spider opened
2017-01-01 07:54:01 [logstats.py] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-01 07:55:44 [monitor_utils.py] INFO: Getting the latest iteration for merchant ariika

I am trying to get the duration between the first and last lines.

here's my logstash configuration:

input {
    tcp {
        port => 5000
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
        }
    }
}


filter {

		grok{
    			match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} WARNING: enable_proxy must %{GREEDYDATA:task_id}"]
  				add_tag => [ "taskStarted" ]
  		}

  		grok{
  				match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} INFO: Getting the latest iteration for %{GREEDYDATA:task_id}"]
  				add_tag => [ "taskTerminated"]
		}

		grok{
				match => [ "message", "%{DATE_EU:timestamp}" ]
			}
			
		date{
		    	match => [ "timestamp", "yy-MM-dd" ]
		   		target => "@timestamp"
			}

		elapsed{
    			start_tag => "taskStarted"
    			end_tag => "taskTerminated"
    			unique_id_field => "task_id"
    			new_event_on_match => true
  		}
	}


output {
	if "_grokparsefailure" not in [tags]{
		stdout {
		codec => rubydebug
		}
         } 
 }
5

You are not converting the full date time text into a timestamp - the event should have a_pateparsefailure tag.
Also, the date filter uses @timestamp as a target by default so no need to specify the target.

Use the inbuilt ISO8601 date filter support.

            date{
		    	match => [ "timestamp", "ISO8601" ]
			}
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.