I have documents being parsed through logstash and sent to elasticsearch. Each document has a date field called delivery. There is another field that has an ID for events that happen throught the day. What I want to do is have these documents go through logstash and somehow find the difference between the first delivery time and the last delivery time for each of these event IDs and have is indexed into elasticsearch as a separate event and index. Is this possible?
Hi!
I have no clear solution that matches your use case, I'll add some ideas from the easiest to the more abstract and complex solution.
Do your documents have some content that allow to identify the first and the last one from a given id?
Something like
timestamp - blablabla - id="abc" message="Starting process blablabla"
... later on ...
timestamp - blablabla - id="abc" messate="End process blablabla"
you could mark these documents with tags and use the elapsed filter to calculate time difference between them. That would be ideal 
If you cannot distinguish the first and last documents from the remaining ones, this will not work.
You can use also elasticsearch filter to get the timestamp of the 1st document of the current day for the given id and use it to calculate the time difference. But doing it for every document... I don't dare to estimate the decrease in performance 
but it may be acceptable depending on the document volume and resources.
I guess you could also schedule alerts as "batch jobs" to be executed at the end of the day, make some kind of aggregation by id to get the oldest and newest documents, and perform some kind of calculations and results indexing or document updating. Either that or do that same query + processing + result output by querying ES from the outside; anyway these solutions would be more laborious.
Maybe another colleague from the community can share more ideas 
You might be able to do something using an aggregate filter (see example 3 in the documentation). If IDs are re-used the next day then this probably will not work.
Ok i will mention also that I need the difference in time for each ID field that I have for not the timestamp but for another date field.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.