Why filebeat pipelines disappoint or SIEM missing authentication patterns

Elastic Security SIEM
,
1

Filebeat has a lot of modules and I like it but I can see that its approach to log parsing is chaotic. Almost all tests I can find for testing ingest pipelines on github missing example of messages about successful or failed user authentication and messages about logout too. However all applications have such messages and if elastic wants its SIEM be useful it should be able to recognize these messages and parse it appropriately.

Let’s take an oracle module for example. If you use Oracle you will not get SIEM notification that someone bruteforce your database and these messages will not be parsed, tagged or categorized. (Filebeat Oracle ingest pipeline can't parse some messages described in oracle docs · Issue #30975 · elastic/beats · GitHub)

If you think that oracle module is exception check Juniper, Postgres or MSSQL module (Filebeat MSSQL ingest pipeline can't parse authentication messages · Issue #31933 · elastic/beats · GitHub)

Seems like current filebeat module tests a bunch of random messages.

Beat tests doesn’t have information about application version that has been tested or it’s compatibility.

“Criticise if you disagree, suggest if you criticise, do/act if you suggest, take responsibility if you do!”

My suggestion is improve tests and documentation by adding authentication tests and its results.

Module documentation should inform users is it parsing login/failed login/logout actions or not cause it’s a core for SIEM. Also these message types should be tagged and categorized appropriately (Filebeat module RabbitMQ - add ECS authentication fields for SIEM · Issue #31159 · elastic/beats · GitHub)

IMHO it’s not a big deal to cover authentication messages for popular apps like Jira/Confluence/Gitlab/Jenkins/Nexus/ESXi/vCenter/Asterisk/Zabbix/ or hardware like Mikrotik/NetApp/PDU/… at least we made it in one company for all apps and hardware.

We will be glad to share our patterns if someone willing to help us made filebeat modules cause we use logstash.

P.S. When I decided to use oracle module I have been stunned when got an error message and discovered that it’s failed cause my date has 2 digits instead of one like in tests (Oct 7). Thanks to ishleenk17 this issue has been fixed (Filebeat Oracle ingest pipeline can't parse date if it's represented by two digit month day · Issue #30974 · elastic/beats · GitHub) I’m still wondering how is it possible that for more than one year it hasn’t been discovered.

P.P.S My respect to the elastic team and their work! A lot done and more to do.

1 Like
2

Hi @Alexander_A! Thanks for taking the time to write up your thoughts on our integrations. It's through feedback like this that we learn and improve. While we generally aim to include broad coverage of event types for each integration (including authentication events), as you've identified there may be times where we don't support some events, most commonly due to a lack of sample data.

Raising GitHub issues and sharing the sample events with us is a tremendous help and we will ensure to update our pipelines based on the issues you've raised.

Totally agree on your documentation suggestion also. As it happens, we are working through documentation guidelines (here) and part of that efforts includes better guidance around which event types are supported by each integration.

On a side note, you mention you are using Filebeat as you use Logstash. Elastic Agent now includes a Logstash output which may unblock you from adopting agent. Most of our new integrations and enhancements to existing integrations are only available with agent, so I'd encourage you to adopt agent if at all possible.

1 Like
3

Hi @jamie.hynds! Thanks for quick reply. As I know elastic agent and fleet agent don't support kafka output. We have a big infrastructure and depend on kafka. As I can see fleet has more patterns than classic filebeat. Does it mean new patterns are not supported by filebeat?

4

What about ECS authentication fields for SIEM? How using event fields I can find all successful/unsuccessful logins or logout made by user or cause of timeout?

From my point of view current modules do not correspond with ECS guides and event category/action/type/outcome fields are missing.

If I’m correct the following event fields should be used:

Successful login:

  • event.category = "authentication"
  • event.action = "logged-in”
  • event.outcome = "success"

Failed login:

  • event.category = "authentication"
  • event.action = "logon-failed"
  • event.outcome = "failure"

User logout - ?

User logout cause of timeout - ?

Let’s check elastic agent postgresql module (
integrations/test-postgresql-13-csv.log-expected.json at main · elastic/integrations · GitHub)

Successful login

message pattern: “connection authorized”

  • event.category = “database”
  • event.type : “info”
  • event.action missing
  • event.outcome missing

Failed login:

message pattern: “authentication failed for user”

  • event.category = "database"
  • event.type : “error”
  • event.action missing
  • event.outcome missing

User logout:

message pattern: “disconnection: session time”

  • event.category = “database”
  • event.type : “info”
  • event.action missing
  • event.outcome missing

Seems there is not way to find authentication event across various modules cause it’s not following ECS guides. May be all modules should be checked for event.category = “authentication” and event.action and covered by auto tests?

5

I decided to find all fleet packages which have event.categoty = “authentication” and have found that only 34 packages from 121 have it. Also I found 111 event.action for event.category = "authentication" and it's not clear why so many and which should be used according to ECS guides.

6

In other SIEM it's clear which rules it has for Cisco ASA from documentation and and what about Elastic SIEM? Seems this part of documentation is missing.
For example we have Cisco ASA but elastic SIEM doesn't inform us about failed logons or blocked users and about new created user it also doesn't inform too. That's why in or company we use logstash to add tags like authentication_success, authentication_failed, logout, user_created, user_deleted and so on for custom SIEM rules. It will be nice to have this info in documentation and guides.

7

No one is disagreeing that some of the integrations, especially the Cisco ones maybe missing some enrichment. The Cisco devices have hundreds of events and the enrichments can only be built off of known samples of events so it's hard to know what u don't know. If u have enrichments/normalizations for missing events u should contribute them back to the repo so everyone can benefit. I don't work for elastic but I contribute what I can instead of just blasting the developers.

1 Like
8

Hello @legoguy1000! Thanks for joining this thread. I can and I will contribute but my post about things like concept, documentation, guides, tests which I can't change and just want to point that it’s should be improved.
P.S. Cisco products have one of the best documentation I have seen and each message has an ID that classify it. IMHO it’s not a big deal to find info how to improve cisco module. There are a lot of open source SIEMs that already did it.

9

No one is arguing that the docs are good and explain every event ID and syntax. It's just a matter of dedicating resources to map out every one due to the large amount of events along with all the other integrations that they're working. I think we all agree that there is absolutely improvement that can be done.

10

Hey @Alexander_A - an Engineer on our team, @kortschak. just submitted a PR to cover some additional authentication events within our ASA integration. If you'd like to take a look and let us know if there's any we've missed, we can look at including them also.

2 Likes
11

@efd6 Thanks you help with Cisco ASA module!
@jamie.hynds as promised I published New detection rules for Cisco ASA - will be glad if you can push it a little :slight_smile:

12

One of the critical things for me when we talk about beat modules/integrations is that I don't know how many SIEM rules there for a module I enabled or does it exist at all.

13

Hi @Alexander_A! This is certainly a common request and it's a workflow we're currently working on improving. @Michelle_Sausa and @Kseniiaign are focusing on initial integrations to rules mappings and may be able to provide some more insight.

14

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.