# How to get running time in this case? **URL:** \ **Category:** Elasticsearch\ **Created:** [August 31, 2019, 4:22pm UTC](https://discuss.elastic.co/t/how-to-get-running-time-in-this-case/197595 "2019-08-31T16:22:45Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![morinooji](https://avatars.discourse-cdn.com/v4/letter/m/edb3f5/32.png) [@morinooji](https://discuss.elastic.co/u/morinooji)\ **Post date:** [August 31, 2019, 4:22pm UTC](https://discuss.elastic.co/t/how-to-get-running-time-in-this-case/197595/1 "2019-08-31T16:22:45Z")
There are two docs which are added by Application Logger (log4j). docs have same instance id (key). #1 { "instance\_id": "ee0e5890-4968-4dec-80c7-69e22ed4001b" , "message": "done", "@TimeStamp" : "2019-08-06T09:33:05.046Z" } #2 { "instance\_id": "ee0e5890-4968-4dec-80c7-69e22ed4001b" , "message": "start", "@TimeStamp" : 2019-08-06T09:00:00.581Z } I can't restructuring the data send to Elasticsearch like below. { "instance\_id": "ee0e5890-4968-4dec-80c7-69e22ed4001b" , "start\_time": "2019-08-06T09:33:05.046Z", "end\_time": "2019-08-06T09:00:00.581Z" } I want to calculate running time between #1 and #2, help me ☹ ---
**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\ **Post date:** [September 4, 2019, 4:33pm UTC](https://discuss.elastic.co/t/how-to-get-running-time-in-this-case/197595/2 "2019-09-04T16:33:36Z")
I think the new [data frame transform](https://www.elastic.co/guide/en/elastic-stack-overview/current/ml-dataframes.html) functionality in Elasticsearch would provide a good solution for you. Using data frame transforms you can do a "group by" on a specific field (`instance_id`) and write the results of an aggregation on that grouped data to another index. In your case you could calculate the running time and write that to another index, which you then could use however you would like to use it. Let's say you had indexed your data like this: ```auto PUT my_index/_doc/1 { "instance_id": "ee0e5890-4968-4dec-80c7-69e22ed4001b", "message": "done", "@TimeStamp": "2019-08-06T09:33:05.046Z" } PUT my_index/_doc/2 { "instance_id": "ee0e5890-4968-4dec-80c7-69e22ed4001b", "message": "start", "@TimeStamp": "2019-08-06T09:00:00.581Z" } ``` You [can define a transform](https://www.elastic.co/guide/en/elasticsearch/reference/current/put-data-frame-transform.html) that aggregates this data with a [scripted metric aggregation](https://www.elastic.co/guide/en/elasticsearch/reference/current/search-aggregations-metrics-scripted-metric-aggregation.html) that calculates the running time, and writes the results to another index `dest_index`: ```auto PUT _data_frame/transforms/transaction_transform { "source": { "index": "my_index" }, "dest": { "index": "dest_index" }, "pivot": { "group_by": { "instance_id": { "terms": { "field": "instance_id.keyword" } } }, "aggregations": { "timestamps": { "scripted_metric": { "init_script": "state.responses = ['start_time':0L,'end_time':0L]", "map_script": """ def message = doc['message.keyword'].value; def timestamp = doc['@TimeStamp'].value; if (message.equals('start')) { state.responses.start_time = timestamp ; } else if (message.equals('done')) { state.responses.end_time = timestamp; } """, "combine_script": "state.responses", "reduce_script": """ def timestamps = ['start_time': 0L, 'end_time': 0L, 'running_time': 0L]; for (responses in states) { timestamps.start_time = responses['start_time']; timestamps.end_time = responses['end_time']; } if (timestamps.start_time != 0L && timestamps.end_time != 0L) { timestamps.running_time = timestamps.end_time.toInstant().toEpochMilli() - timestamps.start_time.toInstant().toEpochMilli(); } return timestamps; """ } } } }, "frequency": "5m", "sync": { "time": { "field": "@TimeStamp", "delay": "60s" } } } ``` The resulting documents in the `dest_index` will now contain a `timestamps.running_time` field with a running time in ms: ```auto "instance_id" : "ee0e5890-4968-4dec-80c7-69e22ed4001b", "timestamps" : { "start_time" : "2019-08-06T09:00:00.581Z", "end_time" : "2019-08-06T09:33:05.046Z", "running_time" : 1984465 ``` ---
**Author:** ![morinooji](https://avatars.discourse-cdn.com/v4/letter/m/edb3f5/32.png) [@morinooji](https://discuss.elastic.co/u/morinooji)\ **Post date:** [September 8, 2019, 6:11am UTC](https://discuss.elastic.co/t/how-to-get-running-time-in-this-case/197595/3 "2019-09-08T06:11:35Z")
Thank you so much!, abdon! ---
**Author:** ![morinooji](https://avatars.discourse-cdn.com/v4/letter/m/edb3f5/32.png) [@morinooji](https://discuss.elastic.co/u/morinooji)\ **Post date:** [September 11, 2019, 11:00am UTC](https://discuss.elastic.co/t/how-to-get-running-time-in-this-case/197595/4 "2019-09-11T11:00:16Z")
this properties occur error... I'm using elasticsearch 7.2.0. "frequeuncy": "5m", "sync": { "time": { "field": "@TimeStamp", "delay": "60s" } } [error message] { "error": { "root\_cause": [ { "type": "x\_content\_parse\_exception", "reason": "[27:3] [data\_frame\_transform\_config] unknown field [frequeuncy], parser not found" } ], "type": "x\_content\_parse\_exception", "reason": "[27:3] [data\_frame\_transform\_config] unknown field [frequeuncy], parser not found" }, "status": 400 } ---
**Author:** ![abdon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/abdon/32/9195_2.png) [@abdon](https://discuss.elastic.co/u/abdon)\ **Post date:** [September 11, 2019, 11:41am UTC](https://discuss.elastic.co/t/how-to-get-running-time-in-this-case/197595/5 "2019-09-11T11:41:50Z")
Looks like you misspelled `frequency` as `frequeuncy`. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 9, 2019, 11:41am UTC](https://discuss.elastic.co/t/how-to-get-running-time-in-this-case/197595/6 "2019-10-09T11:41:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.