Why filebeat pipelines disappoint or SIEM missing authentication patterns

Filebeat has a lot of modules and I like it but I can see that its approach to log parsing is chaotic. Almost all tests I can find for testing ingest pipelines on github missing example of messages about successful or failed user authentication and messages about logout too. However all applications have such messages and if elastic wants its SIEM be useful it should be able to recognize these messages and parse it appropriately.

Let’s take an oracle module for example. If you use Oracle you will not get SIEM notification that someone bruteforce your database and these messages will not be parsed, tagged or categorized. (Filebeat Oracle ingest pipeline can't parse some messages described in oracle docs · Issue #30975 · elastic/beats · GitHub)

If you think that oracle module is exception check Juniper, Postgres or MSSQL module (Filebeat MSSQL ingest pipeline can't parse authentication messages · Issue #31933 · elastic/beats · GitHub)

Seems like current filebeat module tests a bunch of random messages.

Beat tests doesn’t have information about application version that has been tested or it’s compatibility.

“Criticise if you disagree, suggest if you criticise, do/act if you suggest, take responsibility if you do!”

My suggestion is improve tests and documentation by adding authentication tests and its results.

Module documentation should inform users is it parsing login/failed login/logout actions or not cause it’s a core for SIEM. Also these message types should be tagged and categorized appropriately (Filebeat module RabbitMQ - add ECS authentication fields for SIEM · Issue #31159 · elastic/beats · GitHub)

IMHO it’s not a big deal to cover authentication messages for popular apps like Jira/Confluence/Gitlab/Jenkins/Nexus/ESXi/vCenter/Asterisk/Zabbix/ or hardware like Mikrotik/NetApp/PDU/… at least we made it in one company for all apps and hardware.

We will be glad to share our patterns if someone willing to help us made filebeat modules cause we use logstash.

P.S. When I decided to use oracle module I have been stunned when got an error message and discovered that it’s failed cause my date has 2 digits instead of one like in tests (Oct 7). Thanks to ishleenk17 this issue has been fixed (Filebeat Oracle ingest pipeline can't parse date if it's represented by two digit month day · Issue #30974 · elastic/beats · GitHub) I’m still wondering how is it possible that for more than one year it hasn’t been discovered.

P.P.S My respect to the elastic team and their work! A lot done and more to do.

Hi @Alexander_A! Thanks for taking the time to write up your thoughts on our integrations. It's through feedback like this that we learn and improve. While we generally aim to include broad coverage of event types for each integration (including authentication events), as you've identified there may be times where we don't support some events, most commonly due to a lack of sample data.

Raising GitHub issues and sharing the sample events with us is a tremendous help and we will ensure to update our pipelines based on the issues you've raised.

Totally agree on your documentation suggestion also. As it happens, we are working through documentation guidelines (here) and part of that efforts includes better guidance around which event types are supported by each integration.

On a side note, you mention you are using Filebeat as you use Logstash. Elastic Agent now includes a Logstash output which may unblock you from adopting agent. Most of our new integrations and enhancements to existing integrations are only available with agent, so I'd encourage you to adopt agent if at all possible.

Hi @jamie.hynds! Thanks for quick reply. As I know elastic agent and fleet agent don't support kafka output. We have a big infrastructure and depend on kafka. As I can see fleet has more patterns than classic filebeat. Does it mean new patterns are not supported by filebeat?

What about ECS authentication fields for SIEM? How using event fields I can find all successful/unsuccessful logins or logout made by user or cause of timeout?

From my point of view current modules do not correspond with ECS guides and event category/action/type/outcome fields are missing.

If I’m correct the following event fields should be used:

Successful login:

  • event.category = "authentication"
  • event.action = "logged-in”
  • event.outcome = "success"

Failed login:

  • event.category = "authentication"
  • event.action = "logon-failed"
  • event.outcome = "failure"

User logout - ?

User logout cause of timeout - ?

Let’s check elastic agent postgresql module (
integrations/test-postgresql-13-csv.log-expected.json at main · elastic/integrations · GitHub)

Successful login

message pattern: “connection authorized”

  • event.category = “database”
  • event.type : “info”
  • event.action missing
  • event.outcome missing

Failed login:

message pattern: “authentication failed for user”

  • event.category = "database"
  • event.type : “error”
  • event.action missing
  • event.outcome missing

User logout:

message pattern: “disconnection: session time”

  • event.category = “database”
  • event.type : “info”
  • event.action missing
  • event.outcome missing

Seems there is not way to find authentication event across various modules cause it’s not following ECS guides. May be all modules should be checked for event.category = “authentication” and event.action and covered by auto tests?

I decided to find all fleet packages which have event.categoty = “authentication” and have found that only 34 packages from 121 have it. Also I found 111 event.action for event.category = "authentication" and it's not clear why so many and which should be used according to ECS guides.