A new field with a value from the difference between the two existing fields

Elastic Stack Kibana
1

Hello.
Please tell me the direction to the solution.
There is a service that generates records. The values include: fields with a unique object id, a record of the object's action and time.
Question: how to get new fields showing the difference in the values of the existing fields from different records with the same id?
What tools can be used to implement this?
When creating dashboards? Or maybe by means of Elasticsearch itself?
Thank you in advance!

Annotation 2020-08-04 111218
Annotation 2020-08-04 1112181052×488 36.3 KB

2

Hello @Alex9

Which version of the stack are you using? How are you ingesting your data?

Ideally we would have a single document for that transaction id.

3

Hello :slightly_smiling_face:
Elasticsearch version 6.6.1. I receive the data in a script directly into Elasticsearch.
Is it possible to solve the issue using Elasticsearch + Kibana?

But, I also prepared a cluster of version 7.8.1.
From the service, data is sent by a script to the / elk / data directory
Logstash takes the data from the directory and sends it to Elasticsearch.
Current configuration:

input {
  file {
    path => "/elk/data/service*"
    start_position => "beginning"
  }
}
filter {
    csv {
	separator => "|"
	    columns   => ["object_#", "id", "checkpoint", "time"]
    }
    grok {
        match => { "time" => "%{COMBINEDAPACHELOG}"}
    }
    geoip {
        source => "clientip"
    }
    date {
    match => [ "timestamp", "dd/MM/YY HH:mm" ]
    }
    }
output {
stdout { codec => rubydebug }
    elasticsearch {
        hosts => [ "localhost:9200" ]
        user => "elastic"
        password => "my_password..."
        cacert => '/usr/share/logstash/elastic-certificates.pem'
        index => "service2"
        }
}

The data in the catalog /elk/data looks like this:
object 1|BzjZ5P|start|12:30:00 PM
object 1|BzjZ5P|fisnish|12:35:00 PM
object 2|u827Qp|start|1:05:00 PM
object 2|u827Qp|fisnish|1:15:00 PM
object 3|2caUKH|start|1:35:00 PM
object 3|2caUKH|finish|1:39:00 PM
object 4|adaf12|start|1:40:00 PM

Thanks for your answer and time!

4

I'd try using the aggregate filter to get the data you want into a single document. Everything else is much easier this way. https://www.elastic.co/guide/en/logstash/current/plugins-filters-aggregate.html

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.