# Finding time difference between two events with in a single batch of logstash

**URL:** https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813
**Category:** Elasticsearch
**Created:** [August 26, 2019, 4:53pm UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813 "2019-08-26T16:53:42Z")
**Posts on this page:** 7
**Page:** 1

<div class="post-metadata">

### Author: ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)
#### Post date: [August 26, 2019, 4:53pm UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813/1 "2019-08-26T16:53:42Z")

</div>

Hi all...  
Here I am trying to calculate the time difference between two events with the help of input plugin file, filter plugin ruby and elasticsearch  
here is my configuration

> ```
> input	{
> file{
> path => "/root/message_id.txt"
> start_position => "beginning"
> sincedb_path => "/tmp/duration"
> }
> }
> filter	{
> csv {
> columns => ["date_time", "message_id"]
> separator => ","
> }
> elasticsearch {
> hosts => ["http://192.168.1.194:9200"]
> user => "elastic"
> password => "access"
> index => "sgs11"
> query => 'message_id:"%{[message_id]}"'
> sort => "@timestamp:asc"
> fields => { "@timestamp" => "started" }
> }
> date {
> match => ["started", "ISO8601"]
> target => "started"
> }
> ruby {
> code => "event.set('duration_sec', (event.get('@timestamp') - event.get('started')))"
> }
> }
> output {
> elasticsearch {
> hosts => ["http://192.168.1.194:9200"]
> user => "elastic"
> password => "access"
> index => "sgs11"
> }
> stdout {codec => rubydebug }
> }
> 
> ```

input data "/root/message\_id.txt"

```
    date_time, message_id
    2019-07-26T16:03:43.730Z,<test1>
    2019-07-26T16:03:43.730Z,<test1>
    2019-07-26T16:03:43.730Z,<test2>
    2019-07-26T16:03:43.730Z,<test2>
    2019-07-26T16:03:43.730Z,<test3>
    2019-07-26T16:03:43.730Z,<test3>

```

but here elasticsearch filter is not able to get the index time from the previous event as all events are grouped in single batch of logstash.

is there any possibility to resolve this issue

Thanks

---

<div class="post-metadata">

### Author: ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)
#### Post date: [August 29, 2019, 8:14am UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813/2 "2019-08-29T08:14:38Z")

</div>

I have Updated the my configuration as I missed including CSV filter section previously.

---

<div class="post-metadata">

### Author: ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)
#### Post date: [August 29, 2019, 1:25pm UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813/3 "2019-08-29T13:25:50Z")

</div>

when I run the above code I am getting following WARN and ERROR.

> ```
> [2019-08-29T18:53:19,289][INFO][logstash.filters.elasticsearch] New ElasticSearch filter client {:hosts=>["http://192.168.1.194:9200"]}
> [2019-08-29T18:53:19,355][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
> [2019-08-29T18:53:19,367][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
> [2019-08-29T18:53:19,375][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
> [2019-08-29T18:53:19,398][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
> [2019-08-29T18:53:19,406][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
> [2019-08-29T18:53:19,414][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
> [2019-08-29T18:53:19,430][WARN][logstash.filters.elasticsearch] Failed to query elasticsearch for previous event {:index=>"sgs11", :error=>"[404] {\"error\":{\"root_cause\":[{\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"}],\"type\":\"index_not_found_exception\",\"reason\":\"no such index [sgs11]\",\"resource.type\":\"index_or_alias\",\"resource.id\":\"sgs11\",\"index_uuid\":\"_na_\",\"index\":\"sgs11\"},\"status\":404}"}
> [2019-08-29T18:53:19,457][ERROR][logstash.filters.ruby] Ruby exception occurred: can't convert nil into an exact number
> [2019-08-29T18:53:19,464][ERROR][logstash.filters.ruby] Ruby exception occurred: can't convert nil into an exact number
> [2019-08-29T18:53:19,465][ERROR][logstash.filters.ruby] Ruby exception occurred: can't convert nil into an exact number
> [2019-08-29T18:53:19,471][ERROR][logstash.filters.ruby] Ruby exception occurred: can't convert nil into an exact number
> [2019-08-29T18:53:19,472][ERROR][logstash.filters.ruby] Ruby exception occurred: can't convert nil into an exact number
> [2019-08-29T18:53:19,473][ERROR][logstash.filters.ruby] Ruby exception occurred: can't convert nil into an exact number
> [2019-08-29T18:53:19,473][ERROR][logstash.filters.ruby] Ruby exception occurred: can't convert nil into an exact number
> /root/logstash-7.1.0/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
> 
> ```

---

<div class="post-metadata">

### Author: ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)
#### Post date: [August 29, 2019, 1:34pm UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813/4 "2019-08-29T13:34:29Z")

</div>

It is clear that when the first event **2019-07-26T16:03:43.730Z,** processed by logstash it won't find any record in ES index sgs11 but for the second **2019-07-26T16:03:43.730Z,** logstash should be able to find a record in ES and fetch it's @timestamp value and assign it to **started** field.

But it's not happening like that.

I would like to hear your suggestions on this challenge.

Thank you

---

<div class="post-metadata">

### Author: ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)
#### Post date: [August 30, 2019, 11:32am UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813/5 "2019-08-30T11:32:08Z")

</div>

Any update please

---

<div class="post-metadata">

### Author: ![kabali12345](https://avatars.discourse-cdn.com/v4/letter/k/6a8cbe/32.png) [@kabali12345](https://discuss.elastic.co/u/kabali12345)
#### Post date: [September 9, 2019, 8:14am UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813/6 "2019-09-09T08:14:24Z")

</div>

Hi All...  
I would like to receive any alternative solutions as well

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [October 7, 2019, 8:14am UTC](https://discuss.elastic.co/t/finding-time-difference-between-two-events-with-in-a-single-batch-of-logstash/196813/7 "2019-10-07T08:14:26Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.