A new field with a value from the difference between the two existing fields

[Alex9 (Aleksey Shchuka)]

Hello.
Please tell me the direction to the solution.
There is a service that generates records. The values include: fields with a unique object id, a record of the object's action and time.
Question: how to get new fields showing the difference in the values of the existing fields from different records with the same id?
What tools can be used to implement this?
When creating dashboards? Or maybe by means of Elasticsearch itself?
Thank you in advance!

[mattkime (Matthew Kime)]

Hello @Alex9

Which version of the stack are you using? How are you ingesting your data?

Ideally we would have a single document for that transaction id.

[Alex9 (Aleksey Shchuka)]

Hello
Elasticsearch version 6.6.1. I receive the data in a script directly into Elasticsearch.
Is it possible to solve the issue using Elasticsearch + Kibana?

But, I also prepared a cluster of version 7.8.1.
From the service, data is sent by a script to the / elk / data directory
Logstash takes the data from the directory and sends it to Elasticsearch.
Current configuration:

input {
  file {
    path => "/elk/data/service*"
    start_position => "beginning"
  }
}
filter {
    csv {
	separator => "|"
	    columns   => ["object_#", "id", "checkpoint", "time"]
    }
    grok {
        match => { "time" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
    date {
    match => [ "timestamp", "dd/MM/YY HH:mm" ]
    }
    }
output {
stdout { codec => rubydebug }
    elasticsearch {
        hosts => [ "localhost:9200" ]
        user => "elastic"
        password => "my_password..."
        cacert => '/usr/share/logstash/elastic-certificates.pem'
        index => "service2"
        }
}

The data in the catalog /elk/data looks like this:
object 1|BzjZ5P|start|12:30:00 PM
object 1|BzjZ5P|fisnish|12:35:00 PM
object 2|u827Qp|start|1:05:00 PM
object 2|u827Qp|fisnish|1:15:00 PM
object 3|2caUKH|start|1:35:00 PM
object 3|2caUKH|finish|1:39:00 PM
object 4|adaf12|start|1:40:00 PM

Thanks for your answer and time!

[mattkime (Matthew Kime)]

I'd try using the aggregate filter to get the data you want into a single document. Everything else is much easier this way. https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html