How to exploit rules

[Iroshu]

Dear team,

We are actually playing with the detection of elastic and something seems weird for us.
We have created a Rule and we add an action in order to populate an index using our collector :

We choose all fied found on the documentation in order to have everything and to choose the proper field to work on. It's working fine but we discovered that all important information are stored in json format into the field context alerts and it seems it's not possible to exploit them using stand visualisation.

After some research we found that result are stored into the .siem-signals are properly dispatch into several exploitable fields.

We have also noticed that when using the machine learning solution as the results are stored into index using the json format.

Our solution will to have a logstash to read the information and to push it back using good fields but i think we are missing something here. Can you help us ?

[georgii (Georgii Gorbachev)]

Hey @Iroshu, welcome to the community!

Can you please share what stack version are you on and what are you ultimately trying to achieve?

Detection alerts are Elasticsearch documents that a rule writes to .siem-signals-* or .alerts-security.alerts-* indices (depending on the stack version you use). Every alert is written as a separate document, but the rule might write multiple alerts per run. You shouldn't need to reindex alerts as is to a separate index using an Index Connector -- unless you have specific requirements. It should be possible to build visualizations and dashboards on top of the alerts indices -- just make sure your users have access to them. You can learn more about the RBAC in the docs:

Detections prerequisites and requirements | Elastic Security Solution [8.6] |...

[Iroshu]

Hi @georgii ! Thanks

We are actually using the 7.17.9 version. We would like to push theses informations into another index as we plan to add more stuff with other solutions... And we prefer to avoid playing with the system index.

How we can change that ? Any specific options to put into the connector ?

Thanks for your help

[georgii (Georgii Gorbachev)]

Hey @Iroshu, the way you can use placeholders in a rule action's body is documented here:

Create a detection rule | Elastic Security Solution [8.11] | Elastic

In this doc you can find a few examples, including the one showing how you can iterate over the context.alerts array that contains all detection alerts generated during a rule run.

The mustache syntax which is used in rule action bodies is suitable for generating text documents, but falls short when you need to generate valid JSON.

For example, this would be a valid text payload for sending to a webhook:

{{#context.alerts}}
Detection alert for user: {{user.name}}
{{/context.alerts}}

However, generating a JSON doc is problematic. The only easy thing you can do is string concatenation:

{
  "user_names": "{{#context.alerts}}{{user.name}}, {{/context.alerts}}"
}

Or you could index all generated alerts concatenated into a string field, which I'm not sure would be useful at all:

{
  "alerts": "{{context.alerts}}"
}

Some folks managed to workaround this problem for webhooks, you can read more here:

Elastic Rule Connector sends a String instead of JSON to the Webhook

Hey there! First of all, I'd like to describe my situation. I have an Elastic Rule, that successfully creates the Alerts in the Elastic Security. Now I have configured a connector, that allows me to send those Alerts to my custom webhook. It works, but all Alert data (in JSON format by default) is sent as a string. Then I added the Content-Type header with the value application/JSON, but nothing changed. this is a sample of what I receive at webhook when I use a JSON content-type header. [i…

However, indexing ndjson is not an option for the Index Connector, where we'd have to create a valid document. I'd imagine something like that could work:

{
  "alerts": [
    {{#context.alerts}}
      {{{.}}},
    {{/context.alerts}}
    {"last_alert": "fake_document"}
  ]
}

but the form of the index connector doesn't allow to specify non-valid JSON (related issue) and it seems you can't save a rule with a non-valid action body anymore. UPD: you can save a Webhook action with an invalid body (so you can use the workaround), but you can't do the same with an Index action - this one strictly requires the body to be valid JSON (so the workaround won't work).

FWIW one of our teams was planning to work on adding support for triggering rule actions per each generated alert separately, which might be a good option for reindexing them as is into a separate index. I'm not aware of any timeframes though.

And we prefer to avoid playing with the system index.

Unless you need additional or transformed data in your own index (meaning not the alerts as is), you should feel safe and free to read from the system .alerts-security.alerts-* index. This index is managed by the Security app and alerts are written to it by the app itself, you don't need to configure anything to enable that.

Let me know if this information was helpful or not.

[leandrojmp (Leandro Pereira)]

@georgii

FWIW one of our teams was planning to work on adding support for triggering rule actions per each generated alert separately, which might be a good option for reindexing them as is into a separate index.

Just curious, do you have any public issue tracking this? This is one of the main features that the Kibana Alert is lacking and that is forcing us to stay using a third-party tool, ElastAlert, to trigger our alerts.

[georgii (Georgii Gorbachev)]

@leandrojmp I reached out to my colleagues and found out that this is on the short-term roadmap, but is going to be done as part of another feature called Conditional Actions. There's a public issue for that:

[E&C][RAM] [META] Conditional Actions

We will need to update our forEach alert feature to filter alert using alert as data when conditional actions are defined. We will still use the in memory alert to build our actions (for example our context and state)

As I was told, this item refers to the ability to trigger actions per each generated alert separately.

Also, there's another public issue for adding support for writing multiple documents via the Index connector:

[Actions] ES Index action should be able to iterate over context variables

[georgii (Georgii Gorbachev)]

@Iroshu It turns out there's a recent thread where another user had a similar question about reindexing alerts into a separate index.

Creating Multiple Alert Documents when Alert is Triggered

Hi all, I've got a bit of a unique issue. For the system I am developing, data records will be ingested and compared against thresholds to confirm if values are anomalous. To test out this functionality I've set up an alert that applies a range query to check if a value is acceptable or not. I've set up an index connector, so that when the alert is triggered, a document is written to a specific index. The issue I've encountered is that we require one alert document to match up to each docume…

In that thread, it was suggested that for the time being, the best workaround would be leveraging Logstash pipelines for reindexing alerts elsewhere. Related documentation:

• Creating a Logstash pipeline | Logstash Reference [8.11] | Elastic
• Elasticsearch input plugin | Logstash Reference [8.11] | Elastic

[georgii (Georgii Gorbachev)]

@leandrojmp We now have a dedicated public issue:

[Security Solution] Support the ability to trigger a rule action per alert generated

[leandrojmp (Leandro Pereira)]

@georgii

We now have a dedicated public issue

Thanks, I will track this as without the ability to trigger a rule action per alert we cannot fully use the Kibana Alerts interface.

We were trying to migrate from ElastAlert and use only Kibana Alerts for a long time, but decided to keep ElastAlert because of all the Kibana Alerts limitations.

Hope this change in the near future.

[Iroshu]

Hi @georgii,

Thanks for your feedback and explanation provided. I will give you more details about what we try to achieve.

We are actually testing the elastic solution to see if this on is able to generate alertes to another solution. We try to send information by email and by index connector to see what happens.

Below we generate an event and we would like to receive an alerte with as much as possible information about the log.

POST /sf-generate-events/_doc/?pretty
{
"@timestamp": "2023-03-28T12:08:00",
"device": "Fake-solutions1",
"nomderègle": "i can see yoou1",
"numeroalerte": "123d",
"unpeudecontext": "blablablablablbalbalbad",
"meta1":"meta1",
"meta2":"meta2",
"meta3":"meta3",
"meta4":"meta4",
"meta5":"meta5",
"meta6":"meta6"
}

and rule associated :

We would like to receive (in the index or event by mail) some information about rules conditions and also information about meta1 for exemple.Do you think it's possible ?

Again thanks for your help on this .

[Iroshu]

Hi teams,

We have done more test about this and we found that we are able to retrieve information in the Signal index when we are using a custom query :

Below the sample of information used :

And the result in the .siem-signals*

As soon as we use a threshold alerts we lost all information.

Any ideas ? We are losing the interest of the alerting in elastic without this feature.