# Logstash delta time

**URL:** <https://discuss.elastic.co/t/logstash-delta-time/235199>\
**Category:** Logstash\
**Created:** [June 1, 2020, 3:37pm UTC](https://discuss.elastic.co/t/logstash-delta-time/235199 "2020-06-01T15:37:40Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![griffer98](https://avatars.discourse-cdn.com/v4/letter/g/e8c25b/32.png) [@griffer98](https://discuss.elastic.co/u/griffer98)\
**Post date:** [June 1, 2020, 3:37pm UTC](https://discuss.elastic.co/t/logstash-delta-time/235199/1 "2020-06-01T15:37:40Z")

</div>

I have documents being parsed through logstash and sent to elasticsearch. Each document has a date field called delivery. There is another field that has an ID for events that happen throught the day. What I want to do is have these documents go through logstash and somehow find the difference between the first delivery time and the last delivery time for each of these event IDs and have is indexed into elasticsearch as a separate event and index. Is this possible?

---

<div class="post-metadata">

**Author:** ![andres-perez](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andres-perez/32/136461_2.png) [@andres-perez](https://discuss.elastic.co/u/andres-perez)\
**Post date:** [June 1, 2020, 8:24pm UTC](https://discuss.elastic.co/t/logstash-delta-time/235199/2 "2020-06-01T20:24:10Z")

</div>

Hi!

I have no clear solution that matches your use case, I'll add some ideas from the easiest to the more abstract and complex solution.

Do your documents have some content that allow to identify the first and the last one from a given id?  
Something like

```
timestamp - blablabla - id="abc" message="Starting process blablabla"
... later on ...
timestamp - blablabla - id="abc" messate="End process blablabla"

```

you could mark these documents with tags and use the [elapsed filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elapsed.html) to calculate time difference between them. That would be ideal 🤩

If you cannot distinguish the first and last documents from the remaining ones, this will not work.

You can use also [elasticsearch filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html) to get the timestamp of the 1st document of the current day for the given id and use it to calculate the time difference. But doing it for _every_ document... I don't dare to estimate the decrease in performance 🙈 but it may be acceptable depending on the document volume and resources.

I guess you could also schedule alerts as "batch jobs" to be executed at the end of the day, make some kind of aggregation by id to get the oldest and newest documents, and perform some kind of calculations and results indexing or document updating. Either that or do that same query + processing + result output by querying ES from the outside; anyway these solutions would be more laborious.

Maybe another colleague from the community can share more ideas 🙂

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 1, 2020, 8:36pm UTC](https://discuss.elastic.co/t/logstash-delta-time/235199/3 "2020-06-01T20:36:17Z")

</div>

You might be able to do something using an aggregate filter (see [example 3](https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html#plugins-filters-aggregate-example3) in the documentation). If IDs are re-used the next day then this probably will not work.

---

<div class="post-metadata">

**Author:** ![griffer98](https://avatars.discourse-cdn.com/v4/letter/g/e8c25b/32.png) [@griffer98](https://discuss.elastic.co/u/griffer98)\
**Post date:** [June 2, 2020, 3:34pm UTC](https://discuss.elastic.co/t/logstash-delta-time/235199/4 "2020-06-02T15:34:06Z")

</div>

Ok i will mention also that I need the difference in time for each ID field that I have for not the timestamp but for another date field.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 30, 2020, 3:34pm UTC](https://discuss.elastic.co/t/logstash-delta-time/235199/5 "2020-06-30T15:34:14Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
