Runtime Fields in Detection Rule

Elastic Security
1

Hello,

I created a runtime field which I need to use in a detection rule, but I get no alerts. For the detection rule I used "Custom Query" (KQL) and I made sure to use the correct index pattern (the one where the runtime field is defined). So my question is:

How can I create a detection rule based on a runtime field?

Thank you in advance!

2

Hi there @beijo :wave:

Looks like Runtime Field support within Detections is only partially supported at this time. Runtime Fields can be defined in a few different places (ES Indices, Kibana DataViews/Index Patterns, on Search Requests), and based on this issue it seems Detections still needs to add support for Runtime Fields coming from Kibana DataViews (Index Patterns).

That said, as of this PR landing in 7.14, there should be support for Runtime Fields configured directly on ES Indices, so you could try defining the Runtime Fields directly on the ES Index (docs) and the default merge strategy from that above PR (merge_missing_fields_with_source) should ensure it gets picked up and copied over to your alert.

Let us know if you have any troubles with this flow and I'll see if I can help and if not we'll update the details over on the issue for when it gets implemented.

Cheers!
Garrett

2 Likes
3

Hello @spong

thank you for your fast reply.
Do I understand you correctly and you are saying that when I create the runtime field on the index template, the detection rule should pick that up? I tried that by adding the runtime field via Kibana to the index template but still no alerts.

What I did to test that:

  1. Added the runtime field to the index template via Kibana (Stack Management -> Index Management -> Index Templates -> Manage -> Edit -> Mappings -> Runtime Fields -> Add (name: customfield, type: keyword, Script: emit("test")
  2. Created a detection rule on an index pattern that queries the index above which should now trigger for each event. Detection Rule syntax: customfield: "test"

I think I am still missing some parts here.

  • I am not able to see the field via Kibana discover tab (do I need to change something at advanced settings ?
  • Do I need to adjust any other configuration for the detection rule query to include the runtime field in it's result so it can trigger the alert? (I read Retrieve a runtime field) which mentions the fields param for a query to include runtime fields. I tested that in Dev Tools on my above created field which also didnt work..

If possible, can you provide me a step by step guide or a list on what needs to be configured for making my detection rule work?

Thank you!

4

Sorry for the double post, but I finally got it working :slight_smile:
The solution was adding the runtime field via API request to the index.
Correct me if I'm wrong, but that has to be done as changes to the index template only affect newly created indices (which was not the case for the data I was testing in my environment). However when rollover happens to those indices, which leads to a new index for new data, the mappings from the template should apply. Correct?

For the sake of completeness, here the API Request I used:

PUT <indexname>/_mappings
{
  "runtime": {
    "customfield": {
      "type": "long",
      "script": """
        emit(111)
      """
    }
  }
}
2 Likes
5

Glad to hear it's working @beijo! :slightly_smiling_face:

The solution was adding the runtime field via API request to the index.
Correct me if I'm wrong, but that has to be done as changes to the index template only affect newly created indices (which was not the case for the data I was testing in my environment). However when rollover happens to those indices, which leads to a new index for new data, the mappings from the template should apply. Correct?

Yeah, this seems like the case here since you were adding it to the template, so a rollover would need to be triggered for the new mapping to take affect.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.