Combining Events Based on Common session_id Field

[MakoWish (Eric Christensen)]

Good afternoon, all! I first have to say I am very new to ElasticSearch, so I would appreciate some assistance here.

I have a firewall that sends a few different types of logs for the same event, but in a different category. For instance, if someone connects to a site of ours on port 443, I will see something similar to the following (truncated):

event_type: session_event
src_ip: x.x.x.x
src_port: 16864
src_interface: external
dst_ip: x.x.x.x
dst_port: 443
dst_interface: internal
session_id: 101906080442957

That is fine and dandy, but this does not show what the actual firewall did with the event. The firewall event is a separate log message resembling something like:

event_type: firewall_event
action: blocked
reason: ACL
session_id: 101906080442957

I would like to be able to search for (and display) the combined event details so I can not only see the source and destination details, but whether or not the connection was allowed or blocked by the firewall. All related events share a common "session_id" that I would think we could pivot on, but I am not sure how to do this. Any ideas?

Thank you all in advance for the assistance!

EDIT: To note... this is all in the same index.

Eric

[thomasneirynck]

@MakoWish

Are all your events, of both types (firewall and session) in the same index?

What you could do is make a data-table as a Kibana Visualization.

• use a terms-aggregation on session_id to split rows
• add a top-hit metrics for all other fields you would like to see
  • src_ip
  • dst_ip
  • action
  • ...

If you have a single ``session_event corresponding to a single firewall_event, it should produce a table where each row is a session-id with a column for each of those fields.

fwiw, this is a tough one using Elasticsearch, in the way that data is modeled.

Generally, how users approach this is that they denormalize the data at ingest-time by ensuring each individual document combines the events from both sources.

[MakoWish (Eric Christensen)]

Hi Thomas,

Thank you for the reply! These different events are actually from the same "Untangle" firewall, and yes they are all in the same index.

I will try the terms-aggregation on session_id to split rows.

I am not sure it would be possible to join the messages at ingest. They are coming from the same source, and there are three or more messages per session. Here is a very rudimentary example of what comes through:

session_id: 1, type: "session_start", message: "Connection established from..."
session_id: 2, type: "session_start", message: "Connection established to..."
session_id: 1, type: "session_stats", bytes_received: 640, time_elapsed: 31
session_id: 1, type: "session_close", message: "Connection terminated..."
session_id: 2, type: "session_stats", bytes_sent: 1024, time_elapsed: 65
session_id: 2, type: "session_stats", bytes_sent: 512, time_elapsed: 127
session_id: 2, type: "sessino_close", message: "Connection terminated..."

Is it possible to combine these messages in Logstash? In this example, there are three total events for session_id 1, and four total events for session_id 2. I am thinking your terms-aggregation idea would be easiest to just combine them during search, but that would be fantastic to combine them to a single event at ingest.

Thank you,
Eric