Kibana visualization built with context of other logs

Elastic Stack Kibana
1

Hello -

My team is using ElasticSearch/Kibana for security metric related things and we are having trouble with a specific use case:

When a finding is open, a document is created with alerts.status:open associated to the UUID of a finding.id. When that finding has been closed, another document is created with an updated status of 'resolved' with the same finding.id.

We are having trouble accurately visualizing the total count of OPEN findings - i.e. Unique count of findings.id where there is a document for open but there does not exist a document for resolved.

We tried a KQL query like the following however this doesnt work as expected in every case.

  • findings.id:* AND (alerts.status:open AND NOT alerts.status:resolved)

The goal of this would be to get the unique count of findings.Id where there doesn't exist a document for that findings.Id where status=resolved.

Sample Payload:

    {
    findings.id: "1"
    alerts: {
        status:open
      }
    }

    {
    findings.id: "1"
    alerts: {
        status:resolved
      }
    }


    {
    findings.id: "2"
    alerts: {
        status:open
      }
    }

The count here should be 1 but we get the value of 2 -> unique count findings.ids without the context of the resolved document for findings.id: 1

Does anyone have a solution for this or something similar? Happy to provide clarification where needed or anything that would help point us to the correct solution.

2

I am not sure how to do it.
I would consider maintaining another index just for the last status.

3

Yeah we have thought of workarounds like that however it would be ideal if there was a method to do this with Kibana.

4

For anyone searching for a solution to a similar problem, I believe we have solved this with a scripted metric aggregation. We are still testing the accuracy and will be exploring using this in Vega for representing the final number visually but hopefully this helps those searching for a similar solution:

GET index/_search
{
  "aggs": {
  "count_of_Open": {
  "scripted_metric": {
    "init_script": {"id": "open_closed_init-map"},
    "map_script": {"id": "open_closed_map-script-map"},
    "combine_script":{"id": "open_closed_combine-script-map"},
    "reduce_script": {"id": "open_closed_reduction-map"}
  }
}
},
"size": 0
}

The aggregation above is referencing the stored scripts below. Stored scripts allow for the scripts to be multi line rather than one line:

init script:

"source" : """
  state.open_map = [:];
  state.resolved_map = [:];
"""

Map script:

"source" : """

if (doc['alerts.status.keyword'].value == 'open'){ 
  state.open_map.put(doc['alerts.id.keyword'].value, doc['alerts.status.keyword'].value);
} 

if (doc['alerts.status.keyword'].value == 'resolved'){
  state.resolved_map.put(doc['alerts.id.keyword'].value, doc['alerts.status.keyword'].value);
}

"""

Combine script

"source" : """

return [state.open_map, state.resolved_map];
"""

Reduce script (bulk of the work):

"source" : """
 int count = 0;

for ( i in states[0][0].entrySet()){
  if (!states[0][1].containsKey(i.key)){
    count++;
  }
}
return count;

"""

Hope this helps :slight_smile:

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.