Elastic SIEM. Security rules doesn't work

[Dmitriy_Esin]

Hi all!
I have an issue with the Elastic Security Rules.

I've installed the elastic agents on my target instances:

I can successfully see its data streams:

I've created default enrolment policy for my agents:

Here is my yml file generated by Elastic:

DmitriyEsin/elk_yml/blob/main/elastic-agent.yml

As you see, I've configured the Default policy with system-1, Endpoint Security integration, linux-1, Prebuilt Security Rules. But I can's see Prebuilt Security Rules integration in the yml.

Also, I can't understand why default prebuilt security rules don't work (no succeeded rules):

I will be pleased to receive any help!
Thank you!

[Jonathan_Buttner]

Hey @Dmitriy_Esin thanks for the question!
If you click on the Rule Monitoring tab within Rules, does it show any error messages relating to the prebuilt rules?

Could you also provide some Kibana logs? The Detection Rules should log some warnings and errors if they're running into problems.

[Dmitriy_Esin]

Hi @Jonathan_Buttner thanks for you reply!

Here is the example screenshot of the the prebuilt rule:

I have specific warnings in each prebuilt rule.

Unfortunately I can't find any logs from Kibana in web UI (elastic stack is in kubernetes), but I've found container logs (may be it's wrong logs)

DmitriyEsin/elk_yml/blob/main/kibana-kibana-f6b49655-x6t2c.log

Thank you for your help!

[Dmitriy_Esin]

Any update on this? Thank you!

[Dmitriy_Esin]

Here is nothing to configure

[Jonathan_Buttner]

Hey @Dmitriy_Esin looks like the issues is that the rule isn't seeing data within the index patterns auditbeat-* or logs-endpoint.events.*. Could you check the Stack Management -> Index Management page, enable hidden indices, and then search for auditbeat or events and see if anything comes up that would match those patterns?

Here's an example

If the Docs count is zero or nothing is found it likely means that Elasticsearch hasn't received any events from auditbeat or the elastic endpoint.

[Dmitriy_Esin]

Hi @Jonathan_Buttner !
Yes, unfortunately, I've tried it, but there are no indices that we need

[Dmitriy_Esin]

I know that agents create indices themselves when we use any integration, but there is no Prebuilt Security Rules integration in my elastic-agent.yml and I can't configure it

[Dmitriy_Esin]

thank you for your answer! I'll try to install auditbeat.

[Dmitriy_Esin]

Thank you so much!!!
You saved me!
It works well now!

[Dmitriy_Esin]

@Jonathan_Buttner one more question from me:

All rules are well, and I'm very grateful to you for your assistance, but one rule has a status "warning"

Can I kindly ask you to help to understand how to resolve it?

[Jonathan_Buttner]

I'm glad you got it working! The warnings about the event.ingested are indicating that auditbeat and elastic agent aren't populating the event.ingested field. While it's ideal to have this field populated it's not required. The detection rules will fallback to using the @timestamp field instead. You could add a new pipeline that sets this field like this: https://github.com/elastic/beats/pull/20386/files#diff-c6f5ae0e8e34c2011f1b042924c2e20d59df35ae8d70786ec56970fa77c8e973R3-R5

and configure it to be used in auditbeat: Define processors | Auditbeat Reference [8.11] | Elastic

I believe filebeat populates this field by default but the other beats don't.

For more details on what event.ingested is used for: Create a detection rule | Elastic Security Solution [8.11] | Elastic

Specifically item j:

Timestamp override (optional): Select a source event timestamp field. When selected, the rule’s query uses the selected field, instead of the default @timestamp field, to search for alerts. This can help reduce missing alerts due to network or server outages. Specifically, if your ingest pipeline adds a timestamp when events are sent to Elasticsearch, this avoids missing alerts due to ingestion delays.