Finding time difference between two events with in a single batch of logstash

Elastic Stack Elasticsearch
1

Hi all...
Here I am trying to calculate the time difference between two events with the help of input plugin file, filter plugin ruby and elasticsearch
here is my configuration

input	{
  file{
path => "/root/message_id.txt"
start_position => "beginning"
sincedb_path => "/tmp/duration"
  }
}
filter	{
    csv     {
		columns => ["date_time", "message_id"]
separator => ","
            }
 elasticsearch {
	hosts => ["http://192.168.1.194:9200"]
	user => "elastic"
	password => "access"
	index => "sgs11"
	query => 'message_id:"%{[message_id]}"'
    sort => "@timestamp:asc"
	fields => { "@timestamp" => "started" }
 }
date {
  match => ["started", "ISO8601"]
  target => "started"
}
ruby {
  code => "event.set('duration_sec', (event.get('@timestamp') - event.get('started')))"
}
}
output {
elasticsearch {
hosts => ["http://192.168.1.194:9200"]
user => "elastic"
password => "access"
index => "sgs11"
}
stdout {codec => rubydebug }
}

input data "/root/message_id.txt"

    date_time, message_id
    2019-07-26T16:03:43.730Z,<test1>
    2019-07-26T16:03:43.730Z,<test1>
    2019-07-26T16:03:43.730Z,<test2>
    2019-07-26T16:03:43.730Z,<test2>
    2019-07-26T16:03:43.730Z,<test3>
    2019-07-26T16:03:43.730Z,<test3>

but here elasticsearch filter is not able to get the index time from the previous event as all events are grouped in single batch of logstash.

is there any possibility to resolve this issue

Thanks

2

I have Updated the my configuration as I missed including CSV filter section previously.

3

when I run the above code I am getting following WARN and ERROR.

[2019-08-29T18:53:19,289][INFO ][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>["http://192.168.1.194:9200"]}
[2019-08-29T18:53:19,355][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
[2019-08-29T18:53:19,367][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
[2019-08-29T18:53:19,375][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
[2019-08-29T18:53:19,398][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
[2019-08-29T18:53:19,406][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
[2019-08-29T18:53:19,414][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
[2019-08-29T18:53:19,430][WARN ][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
[2019-08-29T18:53:19,457][ERROR][logstash.filters.ruby    ] Ruby exception occurred: can't convert nil into an exact number
[2019-08-29T18:53:19,464][ERROR][logstash.filters.ruby    ] Ruby exception occurred: can't convert nil into an exact number
[2019-08-29T18:53:19,465][ERROR][logstash.filters.ruby    ] Ruby exception occurred: can't convert nil into an exact number
[2019-08-29T18:53:19,471][ERROR][logstash.filters.ruby    ] Ruby exception occurred: can't convert nil into an exact number
[2019-08-29T18:53:19,472][ERROR][logstash.filters.ruby    ] Ruby exception occurred: can't convert nil into an exact number
[2019-08-29T18:53:19,473][ERROR][logstash.filters.ruby    ] Ruby exception occurred: can't convert nil into an exact number
[2019-08-29T18:53:19,473][ERROR][logstash.filters.ruby    ] Ruby exception occurred: can't convert nil into an exact number
/root/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
4

It is clear that when the first event 2019-07-26T16:03:43.730Z, processed by logstash it won't find any record in ES index sgs11 but for the second 2019-07-26T16:03:43.730Z, logstash should be able to find a record in ES and fetch it's @timestamp value and assign it to started field.

But it's not happening like that.

I would like to hear your suggestions on this challenge.

Thank you

5

Any update please

6

Hi All...
I would like to receive any alternative solutions as well

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.