How to get the List of highlighted fields attached per rule in Alerts flyout in Security Analytics

[Sergie]

For my use case I want to understand, is there an API associated, which gives the highlighted fields section in the alerts flyout in Security Plugin in Kibana.

The highlighted fields which gets generated in the alert flyout are dynamic i.e. triggering of different rules as alerts does shows different fields, I want to understand is there an API which can be used to get these fields for a rule or an index where this rules and highlighted field relationship gets stored.

Kindly help with this.

[Philippe_Oberti]

Hey @Sergie,

Here's an explanation of how Highlighted Fields are retrieved then displayed in the Alerts flyout. It might be more information than you asked (if so I apologize), but the logic is actually not straightforward so I figured I'd provide as much as possible here. If you're not interested into understanding how things work in details, you can skip to the end where I answer more directly your question

---

The logic to decide the list of highlighted fields to show happens in this function.
As you can see there, we aggregate fields from multiple places:

• we have a static list of fields that we always want to display. You can see that list here. These would be fields likehost.name, user.name, rule.name...
• to those we add some fields based off of the event's category (see the switch/case here). The value for the event category is retrieved by looking at the event.category field on the alert.
• to those we add fields based off of the event's code (see this switch/case here). The value for the event code is retrieved by looking at the event.code field.
• finally to those as add fields based off of the rule type (see this switch/case here). The rule type value is retrieved by looking at the kibana.alert.rule.type field.

To all the fields above, we add custom fields that have been added to the rule. These fields can be added when creating or editing a rule (see the document here). This was added in 8.10 (see this PR). More recently we made some improvements allowing bulk editing for rule highlighted fields (see this PR). And even more recently we've made improvements to the UI of the flyout to allow users to edit custom rule highlighted fields directly from the flyout Overview tab itself (see this PR)! Note that the code in the very last PR is still behind a disabled feature flag. We should be enabling it soon.

Now the last piece of the puzzle resides in the flyout display logic (specifically here) where we filter out all the field/value pairs where the value is empty. The reason being that we do not want to overload the UI with tons of fields with no values to display... This is why, when following the logic above you would expect to see 20+ fields, but most of the time you only see a handful in the UI.

---

As you can see the logic to retrieve fields isn't straightforward. To answer your question though, to my knowledge there isn't an API, as the logic that decides which fields are shown in the UI is happening on the client side, and the logic is happening when users open the flyout.

[Sergie]

Thanks a lot @Philippe_Oberti for this detailed explanation. This is great.
I really appreciate it.

Just a query is there an file or something which you have handy which contains the exact kibana fieldnames categorized based on the different conditions (based on category, code) or else I will start figuring them out through the code.

[Philippe_Oberti]

@Sergie unfortunately I built my previous answer off of the only file that - to my knowledge - has the information, and it's directly in the code... The links I provided above are pointing to the exact lines for each section though, so hopefully that will make it a little bit easier for you?
Granted that these links are valid at the time of writing this message... they might be slightly outdated in a few weeks if we make changes to the code

For now that's the best I can do, sorry about that!

[Sergie]

Thanks @Philippe_Oberti, yeah those are really useful
I was able to traverse through the code and get those fields.

Just one more question

these counts related to alert by ancestry, how these are calculated, is it based on some field/combination of fields from the alert payload ?

[Philippe_Oberti]

Sorry for my late reply @Sergie, I was off yesterday!

So, while the logic to retrieve the count of alerts related by ancestry is not complex per say, it's not straightforward either . Let me try to explain the flow:

• this is the component that renders that line in the flyout. That component leverages a series of hooks where we ultimately make a call to a api/endpoint/resolver/tree api. This is the function where the magic happens on the server side.
• The first thing we do is fetching all the ancestors of the document that is visualized in the flyout, by making a request to Elasticsearch with the id of the visualized document and the following (non exhaustive) options:

schema: {
    id: 'process.entity_id',
    parent: 'process.parent.entity_id',
    ancestry: 'process.Ext.ancestry',
    name: 'process.name',
    agentId: 'agent.id'
}
ancestors: 200,
indexPatterns: [ '.alerts-security.alerts-default', 'logs-*' ],

• We then retrieve all the descendants for that same document, which is another request to Elasticsearch with the following options (again not exhaustive):

schema: {
    id: 'process.entity_id',
    parent: 'process.parent.entity_id',
    ancestry: 'process.Ext.ancestry',
    name: 'process.name',
    agentId: 'agent.id'
  },
descendants: 500,
indexPatterns: [ '.alerts-security.alerts-default', 'logs-*' ],

• Once we have these, we process and clean the data up a bit and we now have a list of all unique ancestors and descendants of the document we’re looking at.
• Next and final step is yet another request to ElasticSearch, this time to retrieve all the alerts related to the ancestors and descendants. We massage the data a bit, and this is what is returned to the frontend and used in the flyout as a count value.

I hope this was not confusing. Please let me know if you have other questions!

[Sergie]

Thanks for the response @Philippe_Oberti , it's really great help.
I was off for some days as well so really sorry couldn't reply back soon but this was really helpful again I really appreciate it.

On the similar lines I have last couple of questions.

1. Under threat intelligence overview tab, how can I calculate the different counts mentioned for threat matches detected and fields enriched with threat intelligence
   

   

2. How are the Prevalence fields are shown in the overview, how this is done.