Custom App with authentication log

Elastic Security
1

I have custom app with authentication log(consist of login success or failure activities, and logout activities), can I ingest these log directly into log.system.auth-* ?

After ingest direct to log.system.auth-*, will the prebuild Detection Rules works on those Authentication related rules ?

2

Hello,

No, you should not ingest custom logs into the logs-system.auth data stream, this data stream is used by the System integration in Elastic Agent to get logs from the system and parse it.

It expected specific types of message.

For custom logs you should use a custom data stream depending on how you will get your data and send it to Elasticsearch.

If you provide more context on where is your data and how you plan to get it, it may be possible to give directions on how to do it.

3

Hi Leandro, thanks for the advice on "logs-system.auth".

So far the customer only mentioned to abt "custom app" send authentication log to NXLog, then using NXLog to connect direct elasticsearch APi(send as json format).

Basically, customer want to utilize existing index(Datasource) to run Security Detection rules (related to authentication), they dun want to create custom rule(for their custom app).

But i guess this is not a good way, as i check the existing prebuild Detection rules (relate to authentication), they are mainly used for Windows or Linux platform.

For the proper way, we should have custom app data stream, then duplicate the existing rules(relate to authentication), then modify the query to run on customapp data stream, right ?

4

That's not how things work, if this is a custom data, it should go into a custom data stream, the System data stream is used to have System logs from Windows or Linux, putting custom data in these indices may lead to issues and impact the data being collected from System, if they exist.

The pre-built detection rules are create targeting specific data, the system ones will look for system events from windows or linux.

Yes, you should ingest the data into a custom data stream and then create a custom detection rule looking to this data, you do not need to duplicate existing rule, it may be easier to create from scratch.

5

image
image745×680 63.2 KB

Just to check for this kind of ML job in prebuild detection rules, the setup said required data coming from elastic defend or auditd or system, so by default it does not work for custom app, correct ?

In this case, we can create our own custom ML job to perform on Unusual Login volume &
Brute Force Detection based on customapp data source, correct ?

6

Basically yes, you can create a custom ML job and a custom rule based on this ML.

All pre-built ML Jobs and Detection Rules will use data from know integrations, like System, Audit etc.

Any logs from Custom Applications will require custom ML jobs and custom security rules.

Also keep in mind the ML requires a paid license.

7

OOTB ML jobs:

Spike in Logon Events
Spike in Logon Events from a Source IP
Spike in Failed Logon Events
Unusual Hour for a User to Logon
Unusual Source IP for a User to Logon from
Rare User Logon
Unusual Login Activity

"indices": [

  "auditbeat-*",

  "logs-*",

  "filebeat-*",

  "winlogbeat-*"

\]

i just check all OOTB ML job that is related to authentication, all these ML job uses indices:logs-*, so which mean if i have custom app with logs-customapp.auth index, these ML jobs will work, correct ?

What i try to ask is that, if my logs source is mssql or firewall or proxy log with authentication then OOTB ML job will scan these datasources -> logs-mssql-auth or logs-firewall-auth, if pattern matches, it will return the result.

8

No, this is not guaranteed, each ML job will also look for specific fields with specific values on them, they are built to support data from integrations.

For example the Spike in Logon Events , if you check the documentation it mentioned that this job was built to support a couple of integration with data from Windows systems, it requires this data.

This job will also only look for events where the field event.category has the value authentication and event.outcome has the value success as described here, it also has the field winlog.event_data.LogonType as one of the influencers for the job as described here.

So I would not expect this to work with any other kind of data.

As mentioned in the previous posts, pre-built detection rules and ML jobs expected data from existing integrations.

With an Enterprise license you can also edit Elastic rules and change them to look at your custom indices, but the query needs to match, sometimes is way more easier and requires less work to simple create a custom rule.

9

ok, thank you for the advice.