Indicator Match Detection Rule Not Matched and Mapped to Intel Feeds

[ikbal]

Hi guys,

We are using ES 7.10.1 altogether with Logstash and Kibana.

We have ingested TI feeds from MISP and index named as filebeat and we wanted to map and match it to Zscaler logs.

We have tested tens of times just to map and match to it but it seems like the results didn't match and incorrect. Also, the exported fields from MISP is not shown under detection results. What is only shown was the fields from Zscaler logs.

I followed exactly as what is written from elastic documentation about indicator match; for instance; indice A: url.destination:* and indice B: url.full:* and technically if viewed from Elastic Discover for each of these indice: it will shown exactly as: url.destination: and url.full: .

However, when we get these via indicator match rule creation, it didn't work as expected.

In order to test this out;

I took malicious url from MISP and get it browsed via browser: http://www.laforestaincantata.dog/konto/c7a25/ falls under indice B: url.full and from discover it is shown as url.full: http://www.laforestaincantata.dog/konto/c7a25/

then indice A, url.destination shown as url.destination: http://www.laforestaincantata.dog/konto/c7a25/ whereby this results will be shown once we accessed to it via web browser.

Then, detection logic for indicator match by right should be like this:

Results under threat match also indicates the URL that is not malicious and not coming from MISP tagged as under threat match, also the malicious url that I tested not even shown;

Sidenotes:

a. Under rule creation, I also tested with MISP exported field with/without kql for this detection and no results shown.
b. Under rule creation, I also tested with a combination "or" and "and" condition and not giving accurate results.
c. Even if it shown results, the MISP fields is not shown under threat match and if I take the malicious url from MISP and get it browsed via browser, it is not even giving results under threat match.

[Frank_Hassanabad]

Also, the exported fields from MISP is not shown under detection results. What is only shown was the fields from Zscaler logs.

We are working right now in an upcoming release to enrich the signals/detection results with threat intelligence. Right now we do not have that information being copied over, though.

also the malicious url that I tested not even shown;

If during a rule run, we encounter 100 signals we stop looking for results (max_signals) so we don't flood your system from a mis-configured rule with non-stop signals. I think right now you're probably seeing 100 signals per rule run and then we are stopping before we get to the positive match.

In your query above, if you match against filebeat-* as your indicator index pattern with the query being url.full: *, then the rule will look for all records that have a url.full value which exists and will return those results to try to match against any url.full.

So if you have a lot of url.full values, it will go through each of those and see if they match exactly against url.destination within your Zscaler logs which is the xxxnsslog. For each of those matches up to 100 cap limit it then writes it out.

In your screen shot if you put in the column of url.destination I would expect those url's to match all the values from url.full that you have in the filebeat-* index where url.full: * exists.

If you query filebeat-* with url.full: * from discover or timeline are you getting just the malicious url's you're looking for at the moment that are from your list? Or are you getting a lot of other things too? If you're getting a lot of other urls, then that would explain the quick 100 that are causing the cap and all the false positives you are experiencing.

[ikbal]

@Frank_Hassanabad

If you query filebeat-* with url.full: * from discover or timeline are you getting just the malicious url's you're looking for at the moment that are from your list? Or are you getting a lot of other things too? If you're getting a lot of other urls, then that would explain the quick 100 that are causing the cap and all the false positives you are experiencing.

I am not receiving or seeing the malicious urls that I am looking for under indicator detection threat match results. As I wrote earlier, whenever I tested with malicious url, it's not shown. What I got just a non-malicious urls.

Is there any other way to get this fixed? We don't want to see results with non-malicious url when it is by designed by right should match only what is supposed to be malicious url.

[Frank_Hassanabad]

What I got just a non-malicious urls.

That's where I'm trying to understand what your intel feed looks like first to see if we have a bug that is just now surfacing so I can help find the right people to help get it fixed before the next release.

If you're in dev tools and you do this query:

GET filebeat-*/_search
{
  "query": {
    "exists": {
      "field": "url.full"
    }
  }
}

Do you get back only documents that represent malicious urls that you would expect? Are you getting back lots of duplicate records at the moment as well or are there no duplicates?

[ikbal]

I got the results of the malicious url and duplicate records with the same timestamp. It seems like it is shown for at least 4 results with the same url, same timestamp.

[Frank_Hassanabad]

For duplications, there have been other posts depending on which version of the misp module you have installed which should help you with removing duplicates:

MISP and Elastic Security

I have a use case where I want to index attributes from MISP to Elasticsearch for using the new Threat Matching rule in Elastic Security 7.10. Everything is working fine but the issue is when I try to ingest more attributes from misp along with the new one it also again ingests the old ones creating huge duplicates so I delete the whole index before ingesting attributes from MISP. Is there a way where I can prevent the duplicate events to be ingested again?

When it queries each item from the list with the indicator match query set to:

url.full: *

That will cause some of the issues of duplicates you're seeing.

Outside of removing duplicates, if you add the column of url.destination to your signals table what do you see? Are all those url's false positives, any true positives? Also what is the data type of url.destination vs. url.full?

You can see this information with this from dev tools like so, just replace xxx with the correct name of your source index:

GET xxxnsslog*/_mapping/field/url.destination

I'm interested if that is a keyword or if it is something else.

Double check url.full as well but locally that looks to should using the data type of keyword on my system.

GET filebeat-*/_mapping/field/url.full

# Should return something like this which shows it uses data type keyword
  "filebeat-8.0.0-2020.12.31-000005" : {
    "mappings" : {
      "url.full" : {
        "full_name" : "url.full",
        "mapping" : {
          "full" : {
            "type" : "keyword",
            "ignore_above" : 1024,
            "fields" : {
              "text" : {
                "type" : "text",
                "norms" : false
              }
            }
          }
        }
      }
    }
  },

[ikbal]

@Frank_Hassanabad

GET xxxnsslog*/_mapping/field/url.destination

The duplication only can be seen in discover, and not under indicator match detection rule threat match.

I already fixed the duplication on MISP side and whenever new feed is submitted or publish in MISP, by right it should only feed once. In my case, it didn't ingest the old ones.

I tested under dev tools for url.full: but too bad that I couldn't find exactly as what is shown in the discover index of filebeat whereby the url.full will be shown as url.full: (URL)

On file beat under discover, the url.full will shown as, url.full: (url) but on dev tools, it shown otherwise.
What is shown only the hashes and url with only legitimate domain such as VirusTotal.com. That is why I seen false positive results under threat match.

Is there any other way to get it match for only malicious url in my case?

=========
Results for: GET **nsslog/_mapping/field/url.destination

shown as:

{
"***nsslog" : {
"mappings" : {
"url.destination" : {
"full_name" : "url.destination",
"mapping" : {
"destination" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}

========
Results for: GET filebeat-*/_mapping/field/url.full

Shown as:

{
"filebeat-7.10.1-2021.01.13-000001" : {
"mappings" : {
"url.full" : {
"full_name" : "url.full",
"mapping" : {
"full" : {
"type" : "keyword",
"ignore_above" : 1024,
"fields" : {
"text" : {
"type" : "text",
"norms" : false
}
}
}
}
}
}
}
}

[Frank_Hassanabad]

One mapping has text as a default and the other has keyword as a default:

{
   "***nsslog":{
      "mappings":{
         "url.destination":{
            "full_name":"url.destination",
            "mapping":{
               "destination":{
                  "type":"text", <--- Uses text as default
                  "fields":{
                     "keyword":{
                        "type":"keyword",
                        "ignore_above":256
                     }
                  }
               }
            }
         }
      }
   }
}

vs the other way around with:

{
"filebeat-7.10.1-2021.01.13-000001" {
   "mappings":{
      "url.full":{
         "full_name":"url.full",
         "mapping":{
            "full":{
               "type":"keyword", <-- Uses keyword as default
               "ignore_above":1024,
               "fields":{
                  "text":{
                     "type":"text",
                     "norms":false
                  }
               }
            }
         }
      }
   }
}

You might be better off matching keyword against keyword by using something like this:

url.destination.keyword <--> url.full

Otherwise you might end up with something not expected as text will be analyzed and tokenized vs. keyword. Might still work with the way you're doing your mappings but I would suggest exact matches against keyword.

On file beat under discover, the url.full will shown as, url.full: (url) but on dev tools, it shown otherwise.

I don't know if I'm following this 100% but if you're seeing a lot of url.full coming up from dev tools that you do not want to try to match against then that would be the reason for all the false positives.

What you can do to eliminate them is to tune your KQL used in the Indicator index query to match only those things in your filebeat-* that are part of your indicator fields. You can use something like another exists in addition to your url.full: * if you need to filter down to your indicator lists and avoid any false positives from showing up.

[ikbal]

I created this and not getting any results. Is there a way we can test it out under Dev Tools and map to two indices just to test out?

====Rules Definition=====
Index patterns

***nsslog

Filters

action: Blocked

action: Allowed

Custom query

url.destination.keyword : *

Rule type

Indicator Match

Timeline template

Generic Network Timeline

Indicator index patterns

filebeat-*

Indicator mapping

url.destination.keyword MATCHES url.full

Indicator index query

url.full : *

========
Do take note, I also tried to exclude the filter of action allowed and blocked but seems also failing to receive alerts.

[ikbal]

Hi, any recommendations on how to get this fixed?

[Frank_Hassanabad]

It's tricky to help fix this without additional information.

I tested under dev tools for url.full: but too bad that I couldn't find exactly as what is shown in the discover index of filebeat whereby the url.full will be shown as url.full: (URL)

On file beat under discover, the url.full will shown as, url.full: (url) but on dev tools, it shown otherwise.
What is shown only the hashes and url with only legitimate domain such as VirusTotal.com. That is why I seen false positive results under threat match.

I'm not for sure I follow what is going on here. Are you seeing false positives and you know why you are seeing them and need help filtering those out?

If you're not seeing something immediately showing up it could be that you have correctly setup everything but the matching of the indicators are taking a very long time and could be timing out if your list is very large.

• What is the count of your list? Is it 100s, 1k's, 1 million?
• What is the count of your events every 5 minutes? Is it 100s, 1,000's, millions?
• Can we a query the list such as the top 10?
• Can we get an export of your rule in question?
• Can we get an export of your target index mapping?
• Can we get a sample of your target index data?

The more information we can have, the better we can help out.

[ikbal]

Will get back to you on these request soon. Thanks Frank

[ikbal]

for
q1: 1Million

q2: 100s

q3: Yes can do, for example particular user, malware category etc.

q4: Exported rule:
{​​​​​​"author":,"actions":,"created_at":"2021-03-01T07:27:05.677Z","updated_at":"2021-03-01T07:27:05.677Z","created_by":"elastic","description":"NewTestMISP","enabled":true,"false_positives":,"filters":,"from":"now-360s","id":"8514b260-7a5f-11eb-9cd7-a321b0bfaedf","immutable":false,"index":["XXXnsslog"],"interval":"5m","rule_id":"6d8877f5-6c50-4db2-8c1c-ddc97d2941e1","language":"kuery","license":"","output_index":".siem-signals-default","max_signals":100,"risk_score":21,"risk_score_mapping":,"name":"NewTestMISP","query":"url.destination : * ","references":,"meta":{​​​​​​"from":"1m","kibana_siem_app_url":"(URL of ELK"}​​​​​​​,"severity":"low","severity_mapping":[],"updated_by":"elastic","tags":[],"to":"now","type":"threat_match","threat":[],"threat_filters":[],"threat_index":["filebeat-7.11.1-misp"],"threat_query":"url.full : * ","threat_mapping":[{​​​​​​"entries":[{​​​​​​"field":"url.destination","type":"mapping","value":"url.full"}​​​​​​]}​​​​​​],"threat_language":"kuery","throttle":"no_actions","version":1,"exceptions_list":}​​​​​​
{​​​​​​"exported_count":1,"missing_rules":,"missing_rules_count":0}​​​​​​

q5: Whenever we implement the rules. We got message error stated as
" Mar 1, 2021 @ 15:28:34.937
Bulk Indexing of signals failed: Error: Request Timeout after 30000ms name: "NewTestMISP" id: "8514b260-7a5f-11eb-9cd7-a321b0bfaedf" rule id: "6d8877f5-6c50-4db2-8c1c-ddc97d2941e1" signals index: ".siem-signals-default""

q6: Same applies to q5, we can't get because there was an error.

[Frank_Hassanabad]

Bulk Indexing of signals failed: Error: Request Timeout after 30000ms name: "NewTestMISP" id: "8514b260-7a5f-11eb-9cd7-a321b0bfaedf" rule id: "6d8877f5-6c50-4db2-8c1c-ddc97d2941e1" signals index: ".siem-signals-default""

The timeout at the moment is to help prevent incredibly large lists and running times or an underpowered Elastic instance.

Is there a way you can split that rule into smaller rules in which they query smaller segments of the list? That would help reduce and improve the performance. I would query based on a date time or some other attribute of your list for each rule and keep the queries under 10k items at a time.

I would also remove any duplicates from the list if you have any duplicates. 1 million list items will turn into 1 million elastic queries when it runs. I don't think you want 1 million running at the same time but rather you want to run a few rules at different intervals and maybe even longer intervals from the default 5 minutes.

Otherwise if you want to improve the speed and timeliness you should still break up the rule into a few rules that query/segment the list(s) but then also increase your amount of elastic nodes to improve things so you do not see the timeouts.

[ikbal]

Hi Frank,

The duplication issues has been resolved since a month ago, and still we got this error.

Referring to underpowered elastic instance, is there a way we can get this improved?

For splitting the rule, how can we do that ? I think the rule we created, it was only one rule and simple ones.

[Frank_Hassanabad]

1 million list items is going to be a lot of list items to query which is why it is showing the timeouts.

Depending on how your list is setup, you can create multiple rules which will query a section of the list such as based on the date times of the list depending on when/how it is updated. Other people have lists that have identifiers or group labels where they create multiple rules that will query one group for 1 rule and then a 2nd rule will query a different label.

Otherwise you could potentially split the list up into smaller lists and then have 1 indicator match rule query 1 smaller list and then a second rule query another list, so on so forth.

I don't know your setup for Elasticsearch but for our cloud instances you can scale it out by adding more Elastic nodes. Otherwise there's a lot of online documentation (which is above me as I am not the expert at scaling large Elastic nodes), where the documentation should show how to tune and increase your search rates which would help reduce the timeouts.

[ikbal]

I see. By the way, do you have a sample of indicator match detection rule and the ruleset results?