Use timestamp from previous log line

Elastic Stack Kibana
1

Hello,

I have several logs with a timestamp field.

I want to be able to group them by periods. That is, if the timestamp of a log is 30 minutes after the timestamp of the previous log, then it's in a different period.

For example:
LogId Timestamp
1 ...T08:30:00
2 ...T08:32:15
3 ...T09:45:01
4 ...T09:57:09
5 ...T13:48:11

For these logs, I want the groups (1 and 2), (3 and 4) and (5).

The condition I want to check is something like

timestamp: [previouslogtimestamp TO previouslogtimestamp+30m].

Since there is the variable "now" to use for the actual timestamp, I wonder if there's anything similar for the previous logs.

2

Hi Tomás, normally you'd want to use Logstash to do something like this. Unfortunately, I just checked with the Logstash team and it looks like Logstash only processes events individually (or sometimes as a batch). Either way, you can't refer to previous events the way you're asking.

However, I think you can group your events according to "time buckets". Say, grouping them by every hour. So you'd still end up with groups (1 and 2), (3 and 4), and (5). Would this help address your requirement?

Thanks,
CJ

1 Like
3

Hi CJ,

Thanks for you reply. Yes, I've explored that option but it doesn't fit the requirements I want.

If I group them by pre-defined time intervals, the logs won't be grouped by sessions.
For example, if I group them hourly:

LogId Timestamp
1 ...T08:58:32
2 ...T09:01:15

These 2 logs will be grouped into different sessions even though they belong to the same one :frowning:

The end goal is to show, for each IP address, all the LogId's separated by session.
I have this:

ClientIP---LogId---Timestamp

1.2.3.4---1---...T08:58:32
1.2.3.4---2---...T09:01:15

1.2.3.5---1---...T08:59:32
1.2.3.5---2---...T09:02:15

Would it be possible to somehow create an index from the existing one?
This way, I could:
. Group all the events by ClientIp
. Order by Timestamp
. Sequentially, go through each log and store the timestamp of each log as a field SessionEnd of the aggregation
. Therefore, I could compare the timestamp of each log to (SessionEnd+30minutes) to check whether I should group that log with the existing session or create a new session.

Thanks,
Tomás

4

Your best bet would be to use the aggregate filter. Aggregate filter plugin | Logstash Reference [8.11] | Elastic However, that constrains you to one single threaded logstash

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.