Elapsed filter is not working

Elastic Stack Logstash
1

Hello, I am trying to use Elapsed filter to get the duration between two timestamps, here's a snap of my logfile:

2017-01-01 07:53:44 [utils.py] WARNING: enable_proxy must have atleast one http
2017-01-01 07:53:45 [provider.py] DEBUG: Using access key found in environment variable.
2017-01-01 07:53:50 [engine.py] INFO: Spider opened
2017-01-01 07:54:01 [logstats.py] INFO: Crawled 0 pages (at 0 pages/min), scraped 0 items (at 0 items/min)
2017-01-01 07:55:44 [monitor_utils.py] INFO: Getting the latest iteration for merchant ariika

I am trying to get the duration between the first and last lines.

here's my logstash configuration:

input {
    tcp {
        port => 5000
        codec => multiline {
            pattern => "^%{TIMESTAMP_ISO8601} "
            negate => true
            what => previous
        }
    }
}


filter {

		grok{
    			match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} WARNING: enable_proxy must %{GREEDYDATA:task_id}"]
  				add_tag => [ "taskStarted" ]
  		}

  		grok{
  				match => ["message", "%{TIMESTAMP_ISO8601} %{NOTSPACE} INFO: Getting the latest iteration for %{GREEDYDATA:task_id}"]
  				add_tag => [ "taskTerminated"]
		}

		grok{
				match => [ "message", "%{DATE_EU:timestamp}" ]
			}
			
		date{
		    	match => [ "timestamp", "yy-MM-dd" ]
		   		target => "@timestamp"
			}

		elapsed{
    			start_tag => "taskStarted"
    			end_tag => "taskTerminated"
    			unique_id_field => "task_id"
    			new_event_on_match => true
  		}
	}


output {
	if "_grokparsefailure" not in [tags]{
		stdout {
		codec => rubydebug
		}
         } 
 }

my stdout only contains those lines:

logstash_1       | [2017-06-20T15:12:48,787][INFO ][logstash.filters.elapsed ] Elapsed, 'start event' received {:start_tag=>"taskStarted", :unique_id_field=>"task_id"}
logstash_1       | [2017-06-20T15:12:48,848][INFO ][logstash.filters.elapsed ] Elapsed, 'end event' received {:end_tag=>"taskTerminated", :unique_id_field=>"task_id"}
2

There's no task_id in the logs to calculate on that I can see?

3

@warkolm So won't I be able to use elapsed filter with my logs in this form? If not, do you have other options in mind to calculate the duration instead of elapsed filter?

4

It's hard because there is no identifier to link these all together.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.