Display data from collection1 not in collection2

Elastic Stack Kibana
1

Hello,

I have events of different types for example: registered|in_transit|processed
These events are correlated by an ID.

If an event is "registered" but the "in_transit" event is not yet in elastic, I would like the ID's of these "in_transit" events to be displayed.

Would it be possible to display the above situation in Kibana?
If this is not possible in Kibana what other solutions are there?

Any help would be greatly appreciated!

Regards Benny

2

Hi @bterhorst,

I can't think of a way to do that kind of correlation across documents in Kibana right now, because it only supports a subset of the pipeline aggregations available in Elasticsearch.

If you create your own visualization tool you could use a terms aggregation on the ids followed by a top_hits aggregation that includes your type field. Then you could use that in a bucket_selector pipeline aggregation to filter out all id buckets that have both the "registered" and " in_transit" types.

3

Hi Felix,

Thank you for your reply. I will certainly look in to that.

Would it be good practice to make a call to elastic and retrieve the corresponding "registered" event when the "in_transit" event arrives. I could then add a field to it in order to identify it. This way if it does not have that field I know the "in_transit" in not yet present.
Or would you advise against this approach?

Regards benny

4

Yes, doing this during ingestion is definitely a possibility and provides the best performance. If you are using Logstash to ingest your data the Elasticsearch filter could help you with that.

5

Okay ....thanks for the quick feedback!

Regards benny

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.