# ElasticSearch Transform Find Time Difference across documents

**URL:** <https://discuss.elastic.co/t/elasticsearch-transform-find-time-difference-across-documents/242861>\
**Category:** Elasticsearch\
**Created:** [July 28, 2020, 8:19am UTC](https://discuss.elastic.co/t/elasticsearch-transform-find-time-difference-across-documents/242861 "2020-07-28T08:19:56Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![vihar\_mamania](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vihar_mamania/32/72896_2.png) [@vihar\_mamania](https://discuss.elastic.co/u/vihar_mamania)\
**Post date:** [July 28, 2020, 8:19am UTC](https://discuss.elastic.co/t/elasticsearch-transform-find-time-difference-across-documents/242861/1 "2020-07-28T08:19:56Z")

</div>

I have multiple logs being sent out to Elasticsearch in the following way:

```
    {
        "resourceId" : 12345,
        "event" : "SUBMITTED/PROCESSING/SUCCESSFUL"
        "event_time" : "dateString"
    }

```

Resource Id will be same across logs of the same resource.  
What I want to calculate for my metrics is time taken between SUBMITTED -\> SUCCESSFUL steps.

I am using Elastic Transform (since the calculation will be across different documents) for this use-case along with a scripted metric to calculate the time difference,  
I am able to group the resources based on resourceId and do some calculations, but the data I am getting is incorrect, looks like I am not able to handle the data if for same resource the data is stored across different shards

my aggregation looks like below

> ```
> {
> "aggs": {
> "duration": {
> "scripted_metric": {
> "init_script": " state.start = 0; state.end = 0; state.duration = 0",
> "map_script": "if (doc['event.keyword'].value.equals(\"SUBMITTED\")) {state.start = doc.event_time.value.toInstant().toEpochMilli()} else if (doc['event.keyword'].value.equals(\"SUCCESS\")) { state.end = doc['event_time'].value.toInstant().toEpochMilli()} ",
> "combine_script": "if (state.start != 0 && state.end!= 0) {(state.duration = state.end - state.start); return state.duration;} else { state.duration = -1; return state.duration; }",
> "reduce_script": "double b = 0; for (a in states) { if (a != null) { b = a }} return b"
> }
> }
> }
> }
> 
> ```

can someone help me on how this can be achieved?  
Thanks

PS: I don't use Logstash, hence can't use `elapsed()` filter , ideally don't want to use any filters

---

<div class="post-metadata">

**Author:** ![Hendrik\_Muhs](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hendrik_muhs/32/25802_2.png) [@Hendrik\_Muhs](https://discuss.elastic.co/u/Hendrik_Muhs)\
**Post date:** [July 28, 2020, 9:16am UTC](https://discuss.elastic.co/t/elasticsearch-transform-find-time-difference-across-documents/242861/2 "2020-07-28T09:16:48Z")

</div>

The combiner runs per shard, only the reducer runs with all shard results.

I suggest to return `state` in `combine_script` and calculate duration as part of `reduce_script` to cover all states from all shards.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 25, 2020, 9:16am UTC](https://discuss.elastic.co/t/elasticsearch-transform-find-time-difference-across-documents/242861/3 "2020-08-25T09:16:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.