How to exploit rules

Dear team,

We are actually playing with the detection of elastic and something seems weird for us.
We have created a Rule and we add an action in order to populate an index using our collector :

We choose all fied found on the documentation in order to have everything and to choose the proper field to work on. It's working fine but we discovered that all important information are stored in json format into the field context alerts and it seems it's not possible to exploit them using stand visualisation.

After some research we found that result are stored into the .siem-signals are properly dispatch into several exploitable fields.

We have also noticed that when using the machine learning solution as the results are stored into index using the json format.

Our solution will to have a logstash to read the information and to push it back using good fields but i think we are missing something here. Can you help us ?

Hey @Iroshu, welcome to the community! :wave:

Can you please share what stack version are you on and what are you ultimately trying to achieve?

Detection alerts are Elasticsearch documents that a rule writes to .siem-signals-* or .alerts-security.alerts-* indices (depending on the stack version you use). Every alert is written as a separate document, but the rule might write multiple alerts per run. You shouldn't need to reindex alerts as is to a separate index using an Index Connector -- unless you have specific requirements. It should be possible to build visualizations and dashboards on top of the alerts indices -- just make sure your users have access to them. You can learn more about the RBAC in the docs:

Hi @georgii ! Thanks :slightly_smiling_face:

We are actually using the 7.17.9 version. We would like to push theses informations into another index as we plan to add more stuff with other solutions... And we prefer to avoid playing with the system index.

How we can change that ? Any specific options to put into the connector ?

Thanks for your help

Hey @Iroshu, the way you can use placeholders in a rule action's body is documented here:

In this doc you can find a few examples, including the one showing how you can iterate over the context.alerts array that contains all detection alerts generated during a rule run.

The mustache syntax which is used in rule action bodies is suitable for generating text documents, but falls short when you need to generate valid JSON.

For example, this would be a valid text payload for sending to a webhook:

Detection alert for user: {{}}

However, generating a JSON doc is problematic. The only easy thing you can do is string concatenation:

  "user_names": "{{#context.alerts}}{{}}, {{/context.alerts}}"

Or you could index all generated alerts concatenated into a string field, which I'm not sure would be useful at all:

  "alerts": "{{context.alerts}}"

Some folks managed to workaround this problem for webhooks, you can read more here:

However, indexing ndjson is not an option for the Index Connector, where we'd have to create a valid document. I'd imagine something like that could work:

  "alerts": [
    {"last_alert": "fake_document"}

but the form of the index connector doesn't allow to specify non-valid JSON (related issue) and it seems you can't save a rule with a non-valid action body anymore. UPD: you can save a Webhook action with an invalid body (so you can use the workaround), but you can't do the same with an Index action - this one strictly requires the body to be valid JSON (so the workaround won't work).

FWIW one of our teams was planning to work on adding support for triggering rule actions per each generated alert separately, which might be a good option for reindexing them as is into a separate index. I'm not aware of any timeframes though.

And we prefer to avoid playing with the system index.

Unless you need additional or transformed data in your own index (meaning not the alerts as is), you should feel safe and free to read from the system .alerts-security.alerts-* index. This index is managed by the Security app and alerts are written to it by the app itself, you don't need to configure anything to enable that.

Let me know if this information was helpful or not.

1 Like

Just curious, do you have any public issue tracking this? This is one of the main features that the Kibana Alert is lacking and that is forcing us to stay using a third-party tool, ElastAlert, to trigger our alerts.

@leandrojmp I reached out to my colleagues and found out that this is on the short-term roadmap, but is going to be done as part of another feature called Conditional Actions. There's a public issue for that:

We will need to update our forEach alert feature to filter alert using alert as data when conditional actions are defined. We will still use the in memory alert to build our actions (for example our context and state)

As I was told, this item refers to the ability to trigger actions per each generated alert separately.

Also, there's another public issue for adding support for writing multiple documents via the Index connector:

@Iroshu It turns out there's a recent thread where another user had a similar question about reindexing alerts into a separate index.

In that thread, it was suggested that for the time being, the best workaround would be leveraging Logstash pipelines for reindexing alerts elsewhere. Related documentation:

@leandrojmp We now have a dedicated public issue:

Thanks, I will track this as without the ability to trigger a rule action per alert we cannot fully use the Kibana Alerts interface.

We were trying to migrate from ElastAlert and use only Kibana Alerts for a long time, but decided to keep ElastAlert because of all the Kibana Alerts limitations.

Hope this change in the near future.


Hi @georgii,

Thanks for your feedback and explanation provided. I will give you more details about what we try to achieve.

We are actually testing the elastic solution to see if this on is able to generate alertes to another solution. We try to send information by email and by index connector to see what happens.

Below we generate an event and we would like to receive an alerte with as much as possible information about the log.

POST /sf-generate-events/_doc/?pretty
"@timestamp": "2023-03-28T12:08:00",
"device": "Fake-solutions1",
"nomderègle": "i can see yoou1",
"numeroalerte": "123d",
"unpeudecontext": "blablablablablbalbalbad",

and rule associated :

We would like to receive (in the index or event by mail) some information about rules conditions and also information about meta1 for exemple.Do you think it's possible ?

Again thanks for your help on this .

Hi teams,

We have done more test about this and we found that we are able to retrieve information in the Signal index when we are using a custom query :
Below the sample of information used :

And the result in the .siem-signals*

As soon as we use a threshold alerts we lost all information.

Any ideas ? We are losing the interest of the alerting in elastic without this feature.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.