Filebeat MISP module: ECS mapping mistakenly removed?

Elastic Stack Beats
1

Hey people,

In a commit from november, the ability to map domain indicators of compromise to url.domain / dns.question.name was removed from the MISP module. See line 59 in the old file: https://github.com/elastic/beats/commit/54591696452e6d5b0197d7533eb6e322241c6022#diff-a1b037f238f44c4b14e8ffbee423304d

Any particular reason for this? It seems that someone tried to refactor the field renaming into the function "convertFields", but it was only halfway done. As a result, we now have missing mappings in 7.6...

2

These fields are removed on purpose. When the match conditions are combined with an "OR", it would not be proper to set all these fields, since it will be considered as an "AND", which requires all the fields to be matched. The attack_pattern_kql field shows the logic behind this. BTW, when there is only one field or when the fields are combined with an "AND", the fields are set. Thanks.

3

Ahh I think I understand the reasoning for your change.
My argument would be: why not have both?

  • The proper attack pattern with the right logical operators in attack_pattern(_kql)
  • The raw indicators in the appropriate ECS fields, like e.g. url.domain and dns.question.name for indicator type domain.

To illustrate why one could still need the raw indicators, here is a use case:
I have DNS query logs that store domains in the field dns.question.name (millions per day). I also have a MISP-Feed and would like to search for domain IOCs in my query logs. Now, I could use the attack_pattern_kql and issue one query for each IOC (also in the thousands). If I do this regularly, it causes a massive search load on the cluster.
Instead, I coalesce all domain indicators into a dns.question.name array of one document using a transform. Then I run one terms lookup query against the DNS query logs, which comes back in <1s.
This approach is not possible anymore if I don't have the raw domain indicators somewhere, but instead only have pre-written KQL strings.

Leaving the raw indicators in without the KQL logic enables custom use cases, and does not hurt people who do want to rely on the pre-written queries.

4

One more point:

it would not be proper to set all these fields, since it will be considered as an "AND"

Is this an official semantic? Isn't it up to the query to decide whether the individual fields are considered "AND" or "OR"?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.