Combining Events Based on Common session_id Field

Good afternoon, all! I first have to say I am very new to ElasticSearch, so I would appreciate some assistance here.

I have a firewall that sends a few different types of logs for the same event, but in a different category. For instance, if someone connects to a site of ours on port 443, I will see something similar to the following (truncated):

event_type: session_event
src_ip: x.x.x.x
src_port: 16864
src_interface: external
dst_ip: x.x.x.x
dst_port: 443
dst_interface: internal
session_id: 101906080442957

That is fine and dandy, but this does not show what the actual firewall did with the event. The firewall event is a separate log message resembling something like:

event_type: firewall_event
action: blocked
reason: ACL
session_id: 101906080442957

I would like to be able to search for (and display) the combined event details so I can not only see the source and destination details, but whether or not the connection was allowed or blocked by the firewall. All related events share a common "session_id" that I would think we could pivot on, but I am not sure how to do this. Any ideas?

Thank you all in advance for the assistance!

EDIT: To note... this is all in the same index.



Are all your events, of both types (firewall and session) in the same index?

What you could do is make a data-table as a Kibana Visualization.

  • use a terms-aggregation on session_id to split rows
  • add a top-hit metrics for all other fields you would like to see
    • src_ip
    • dst_ip
    • action
    • ...

If you have a single ``session_event corresponding to a single firewall_event, it should produce a table where each row is a session-id with a column for each of those fields.

fwiw, this is a tough one using Elasticsearch, in the way that data is modeled.

Generally, how users approach this is that they denormalize the data at ingest-time by ensuring each individual document combines the events from both sources.

Hi Thomas,

Thank you for the reply! These different events are actually from the same "Untangle" firewall, and yes they are all in the same index.

I will try the terms-aggregation on session_id to split rows.

I am not sure it would be possible to join the messages at ingest. They are coming from the same source, and there are three or more messages per session. Here is a very rudimentary example of what comes through:

session_id: 1, type: "session_start", message: "Connection established from..."
session_id: 2, type: "session_start", message: "Connection established to..."
session_id: 1, type: "session_stats", bytes_received: 640, time_elapsed: 31
session_id: 1, type: "session_close", message: "Connection terminated..."
session_id: 2, type: "session_stats", bytes_sent: 1024, time_elapsed: 65
session_id: 2, type: "session_stats", bytes_sent: 512, time_elapsed: 127
session_id: 2, type: "sessino_close", message: "Connection terminated..."

Is it possible to combine these messages in Logstash? In this example, there are three total events for session_id 1, and four total events for session_id 2. I am thinking your terms-aggregation idea would be easiest to just combine them during search, but that would be fantastic to combine them to a single event at ingest.

Thank you,

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.