Event correlation (with EQL)

Elastic Observability
1

Hello everyone,

I am currently deploying a Elastic, Kibana, Fleet & agents stack for a project.
Details:

  • Elastic (three nodes), in a cluster, version 8.6
  • Kibana (one node), version 8.6
  • Fleet servers (on each node)
  • Elastic Agents

This project requires manual event correlation (before, maybe, going with ML).

I got all my logs from VPN (checkpoint), Windows, Linux, and other stuff needed (each one has an index and data is good)

Typical use cases to correlate are :

  • VPN Login => RDP Windows
  • VPN Login => SSH Linux (Redhat based)
  • VPN Login => ... => and so on

To do so, I used EQL to write some rules. When I try to write a rule for windows, I can see connected users for example.
Here is a rule example (with only Windows index):

sequence 
[authentication where event.action=="logged-in" and event.code =="4624"]
[authentication where user.name like "*.XXX"]
until [authentication where event.action=="logged-out" and event.code == "4634"]

This rule is working, but gives me some system users (afaik, system users does not have the domain suffix in their name)

Now, when I add a rule to get the username of the VPN user to correlate these two events, it says
verification_exception: Found 1 problem line 2:7: Unknown column [checkpoint.username], [...]?
Is this the normal behaviour ?

Note: the data is available and validated, it is working when I set to a smaller index (Windows for example). But I cannot correlate different sources with only one index.

As far as I experienced, I got this error because I'm trying to get some fields on a large index (to get all data logs-* )

Do you have any idea ? Or am I doing it the wrong way ?

Thanks

2

UPDATE: After a lot of research and tests. I figured it out that EQL does not support multiple indexes access for correlation. Right now, it only supports one index.

So, I managed to create a "correlation like" behavior with DSL requests and aggregations. Like this, I was able to filter for some values (will say that it's the "correlation/common key" between events) and get only the latest (in my case, last 5 minutes) logs.

It's not really the correlation that I was expecting, but this is doing the job as I got events with the same key together.