Find newly added Event that have a unique ID

Elastic Stack Kibana
1

Hello, is there any existing feature or way to do this?
I need to see Data that has been newly added (fresh ID) in a specific time frame.

Example:

Time 12:00
Data:
Ex 1 - ID: 1 - Name: ...
Ex 2 - ID: 2 - Name: ...
Ex 3 - ID: 3 - Name: ...

Time 12:05
Data:
Ex 1 - ID: 1 - Name: ...
Ex 2 - ID: 2 - Name: ...
Ex 3 - ID: 3 - Name: ...
Ex 4 - ID: 4 - Name: ...
Ex 5 - ID: 5 - Name: ...

How can I make it, so if i choose the time 12:05 and only want to see the difference of the previous time to see this:

Time 12:05 (Difference)
Data:
Ex 4 - ID: 4 - Name: ...
Ex 5 - ID: 5 - Name: ...

There exists multiple ways to show Serial Diff of events via a unique ID - but it would only show the number 2 in my shown example. I need to see these 2 Ex files with all their attributes and not how many of them are different.
Thank you in advance!

2

You could implement this using a continuous transform. As group_by you can use the id and in the aggregation part use a min(timestamp) to write a first_seen field. Given this field you can query on the dest index for what you are aiming for.

To copy all other fields you can use something like this.

Note: If your dataset is small you might also do this via ordinary search+aggs, but I assume you have larger volumes, that's why I suggest transform.

3

Thank you for pointing me in that direction, i will have a look and try to implement this.
By the sounds of it, this will then timestamp an ID if it has been appeared to be then filtered and searched for.

Is there also a solution for the reverse of it? To see which unique id is not to be seen anymore?

Like this:

Time 12:00
Ex 1 - ID 1 - Name ..
Ex 2 - ID 2 - Name ..
Ex 3 - ID 3 - Name ..

Time 12:05
Ex 1 - ID 1 - Name ..
Ex 3 - ID 3 - Name ..

to get this result ->

Time 12:05(disappeared)
Ex 2 - ID 2 - Name ..

4

Similar concept, if you add an aggregation with max(timestamp) written to a field e.g. last_seen you can query for id's that disappeared.

You can automatically delete those entries using the retention_policy feature, however than you won't be able to query for disappeared id's obviously.

1 Like
5

Thank you for now, I will get back to you to tell you how it went! :grinning:

6

Thank you, This solved my original question.
It seems like It's not the full solution that I need and I need to look deeper into this and research myself first.
Thank you again for taking your time to help me!

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.