How to get the List of highlighted fields attached per rule in Alerts flyout in Security Analytics

[Sergie]

For my use case I want to understand, is there an API associated, which gives the highlighted fields section in the alerts flyout in Security Plugin in Kibana.

The highlighted fields which gets generated in the alert flyout are dynamic i.e. triggering of different rules as alerts does shows different fields, I want to understand is there an API which can be used to get these fields for a rule or an index where this rules and highlighted field relationship gets stored.

Kindly help with this.

[Philippe_Oberti]

Hey @Sergie,

Here's an explanation of how Highlighted Fields are retrieved then displayed in the Alerts flyout. It might be more information than you asked (if so I apologize), but the logic is actually not straightforward so I figured I'd provide as much as possible here. If you're not interested into understanding how things work in details, you can skip to the end where I answer more directly your question

---

The logic to decide the list of highlighted fields to show happens in this function.
As you can see there, we aggregate fields from multiple places:

• we have a static list of fields that we always want to display. You can see that list here. These would be fields likehost.name, user.name, rule.name...
• to those we add some fields based off of the event's category (see the switch/case here). The value for the event category is retrieved by looking at the event.category field on the alert.
• to those we add fields based off of the event's code (see this switch/case here). The value for the event code is retrieved by looking at the event.code field.
• finally to those as add fields based off of the rule type (see this switch/case here). The rule type value is retrieved by looking at the kibana.alert.rule.type field.

To all the fields above, we add custom fields that have been added to the rule. These fields can be added when creating or editing a rule (see the document here). This was added in 8.10 (see this PR). More recently we made some improvements allowing bulk editing for rule highlighted fields (see this PR). And even more recently we've made improvements to the UI of the flyout to allow users to edit custom rule highlighted fields directly from the flyout Overview tab itself (see this PR)! Note that the code in the very last PR is still behind a disabled feature flag. We should be enabling it soon.

Now the last piece of the puzzle resides in the flyout display logic (specifically here) where we filter out all the field/value pairs where the value is empty. The reason being that we do not want to overload the UI with tons of fields with no values to display... This is why, when following the logic above you would expect to see 20+ fields, but most of the time you only see a handful in the UI.

---

As you can see the logic to retrieve fields isn't straightforward. To answer your question though, to my knowledge there isn't an API, as the logic that decides which fields are shown in the UI is happening on the client side, and the logic is happening when users open the flyout.

[Sergie]

Thanks a lot @Philippe_Oberti for this detailed explanation. This is great.
I really appreciate it.

Just a query is there an file or something which you have handy which contains the exact kibana fieldnames categorized based on the different conditions (based on category, code) or else I will start figuring them out through the code.

[Philippe_Oberti]

@Sergie unfortunately I built my previous answer off of the only file that - to my knowledge - has the information, and it's directly in the code... The links I provided above are pointing to the exact lines for each section though, so hopefully that will make it a little bit easier for you?
Granted that these links are valid at the time of writing this message... they might be slightly outdated in a few weeks if we make changes to the code

For now that's the best I can do, sorry about that!

[Sergie]

Thanks @Philippe_Oberti, yeah those are really useful
I was able to traverse through the code and get those fields.

Just one more question

these counts related to alert by ancestry, how these are calculated, is it based on some field/combination of fields from the alert payload ?

[Philippe_Oberti]

Sorry for my late reply @Sergie, I was off yesterday!

So, while the logic to retrieve the count of alerts related by ancestry is not complex per say, it's not straightforward either . Let me try to explain the flow:

• this is the component that renders that line in the flyout. That component leverages a series of hooks where we ultimately make a call to a api/endpoint/resolver/tree api. This is the function where the magic happens on the server side.
• The first thing we do is fetching all the ancestors of the document that is visualized in the flyout, by making a request to Elasticsearch with the id of the visualized document and the following (non exhaustive) options:

schema: {
    id: 'process.entity_id',
    parent: 'process.parent.entity_id',
    ancestry: 'process.Ext.ancestry',
    name: 'process.name',
    agentId: 'agent.id'
}
ancestors: 200,
indexPatterns: [ '.alerts-security.alerts-default', 'logs-*' ],

• We then retrieve all the descendants for that same document, which is another request to Elasticsearch with the following options (again not exhaustive):

schema: {
    id: 'process.entity_id',
    parent: 'process.parent.entity_id',
    ancestry: 'process.Ext.ancestry',
    name: 'process.name',
    agentId: 'agent.id'
  },
descendants: 500,
indexPatterns: [ '.alerts-security.alerts-default', 'logs-*' ],

• Once we have these, we process and clean the data up a bit and we now have a list of all unique ancestors and descendants of the document we’re looking at.
• Next and final step is yet another request to ElasticSearch, this time to retrieve all the alerts related to the ancestors and descendants. We massage the data a bit, and this is what is returned to the frontend and used in the flyout as a count value.

I hope this was not confusing. Please let me know if you have other questions!

[Sergie]

Thanks for the response @Philippe_Oberti , it's really great help.
I was off for some days as well so really sorry couldn't reply back soon but this was really helpful again I really appreciate it.

On the similar lines I have last couple of questions.

1. Under threat intelligence overview tab, how can I calculate the different counts mentioned for threat matches detected and fields enriched with threat intelligence
   

   

2. How are the Prevalence fields are shown in the overview, how this is done.
   

[Philippe_Oberti]

Hey @Sergie , sorry for my even more late reply this time, I've been pretty sick over the last week...

So, be ready because this is another complex one!

These 2 numbers - while in the same section and displayed bascially next to each other - are not calculated the same way at all.
In summary, both want to highlight a relation between the visualized alert and some threat intelligence related data. While the threat matches detected value is a persistent value that happens when the alert is created, the value for fields enriched with threat intelligence is not persistent and calculated when we open the flyout.

---

Let me explain each a bit more in details:

1. Threat match detected

This value represents the number of documents related to the alert by a certain field. Let's take an example here: I'm going to create 2 indices then run an Indicator match rule to create alerts that will have the correct values to show something in the threat match detected section.

First create a new index, set it's mapping and add a basic document to it:

PUT test-index

PUT test-index/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    },
    "file": {
        "properties": {
          "hash": {
            "properties": {
              "md5": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "sha1": {
                "type": "keyword",
                "ignore_above": 1024
              },
              "sha256": {
                "type": "keyword",
                "ignore_above": 1024
              }
            }
          }
        }
    }
  }
}

POST test-index/_doc
{
  "@timestamp": 1748376265490,
  "file.hash.sha256":"abcdef"
}

Then I'm going to create a second index, set its mapping and add a document that will match the value in the previously added document:

PUT threat-index

PUT threat-index/_mapping
{
  "properties": {
    "@timestamp": {
      "type": "date"
    }
  }
}

POST threat-index/_doc
{
  "@timestamp": 1748376265490,
  "threat.indicator.file.hash.sha256": "abcdef"
}

Now if I create a new Indicator Match rule that match my file.hash.sha256 field with the threat.indicator.file.hash.sha256 field (you can find all the field pairs here, this will generate one alert that will be enriched with the following fields:

You can see on that picture that that alert has a threat.enrichments field with the following information:

• the index of where the threat documents are
• the field that the alert matched against (in our case file.hash.sha256)
• the value of the field (in our case abcdef)

If you expand the flyout (with the Expand details button in the top left corner, or by clicking on the Threat Intelligence title of the section) you will see the following more detailed information:

Most of the logic for this section is in this hook. You'll see that first we look at which fields (that we call eventFields in the code) we care about here (that will match the array I provided as a link above), and the values are returned.

---

2. Enriched with threat intelligence

This section is done in real time, meaning the value are calculated when you open the flyout. In the same hook linked right above, we also make a query to fetch some more data.
That query is:

• looking at the indices entered in the securitySolution:defaultThreatIndex, which you can access via the Advanced Settings under Security Solution -> Threat indices (by default you should just have logs-ti_*)
  
  
• by default fetching for the last 30 days to now. This is what is always used in the overview tab of the flyout. If you uses the expanded section of the flyout, you have a date picker that you can use whatever timeframe you want (just applied to the expanded section though).
• using the same eventFields used in the previous section

---

Just FYI, before presenting the results, we're removing all the duplicates to make sure the same document will not appear in both sections.

I hope this helps! If you need more details I might have to reach out to the team that manages this as this is basically the extend of my knowledge
Let me know!!