Indicator Match Detection Rule Not Matched and Mapped to Intel Feeds

Hi guys,

We are using ES 7.10.1 altogether with Logstash and Kibana.

We have ingested TI feeds from MISP and index named as filebeat and we wanted to map and match it to Zscaler logs.

We have tested tens of times just to map and match to it but it seems like the results didn't match and incorrect. Also, the exported fields from MISP is not shown under detection results. What is only shown was the fields from Zscaler logs.

I followed exactly as what is written from elastic documentation about indicator match; for instance; indice A: url.destination:* and indice B: url.full:* and technically if viewed from Elastic Discover for each of these indice: it will shown exactly as: url.destination: and url.full: .

However, when we get these via indicator match rule creation, it didn't work as expected.

In order to test this out;

I took malicious url from MISP and get it browsed via browser: http://www.laforestaincantata.dog/konto/c7a25/ falls under indice B: url.full and from discover it is shown as url.full: http://www.laforestaincantata.dog/konto/c7a25/

then indice A, url.destination shown as url.destination: http://www.laforestaincantata.dog/konto/c7a25/ whereby this results will be shown once we accessed to it via web browser.

Then, detection logic for indicator match by right should be like this:
Screenshot 2021-01-28 at 12.13.44 PM

Results under threat match also indicates the URL that is not malicious and not coming from MISP tagged as under threat match, also the malicious url that I tested not even shown;

Sidenotes:

a. Under rule creation, I also tested with MISP exported field with/without kql for this detection and no results shown.
b. Under rule creation, I also tested with a combination "or" and "and" condition and not giving accurate results.
c. Even if it shown results, the MISP fields is not shown under threat match and if I take the malicious url from MISP and get it browsed via browser, it is not even giving results under threat match.

Also, the exported fields from MISP is not shown under detection results. What is only shown was the fields from Zscaler logs.

We are working right now in an upcoming release to enrich the signals/detection results with threat intelligence. Right now we do not have that information being copied over, though.

also the malicious url that I tested not even shown;

If during a rule run, we encounter 100 signals we stop looking for results (max_signals) so we don't flood your system from a mis-configured rule with non-stop signals. I think right now you're probably seeing 100 signals per rule run and then we are stopping before we get to the positive match.

In your query above, if you match against filebeat-* as your indicator index pattern with the query being url.full: *, then the rule will look for all records that have a url.full value which exists and will return those results to try to match against any url.full.

So if you have a lot of url.full values, it will go through each of those and see if they match exactly against url.destination within your Zscaler logs which is the xxxnsslog. For each of those matches up to 100 cap limit it then writes it out.

In your screen shot if you put in the column of url.destination I would expect those url's to match all the values from url.full that you have in the filebeat-* index where url.full: * exists.

If you query filebeat-* with url.full: * from discover or timeline are you getting just the malicious url's you're looking for at the moment that are from your list? Or are you getting a lot of other things too? If you're getting a lot of other urls, then that would explain the quick 100 that are causing the cap and all the false positives you are experiencing.

I am not receiving or seeing the malicious urls that I am looking for under indicator detection threat match results. As I wrote earlier, whenever I tested with malicious url, it's not shown. What I got just a non-malicious urls.

Is there any other way to get this fixed? We don't want to see results with non-malicious url when it is by designed by right should match only what is supposed to be malicious url.

What I got just a non-malicious urls.

That's where I'm trying to understand what your intel feed looks like first to see if we have a bug that is just now surfacing so I can help find the right people to help get it fixed before the next release.

If you're in dev tools and you do this query:

GET filebeat-*/_search
{
  "query": {
    "exists": {
      "field": "url.full"
    }
  }
}

Do you get back only documents that represent malicious urls that you would expect? Are you getting back lots of duplicate records at the moment as well or are there no duplicates?

I got the results of the malicious url and duplicate records with the same timestamp. It seems like it is shown for at least 4 results with the same url, same timestamp.

For duplications, there have been other posts depending on which version of the misp module you have installed which should help you with removing duplicates:

When it queries each item from the list with the indicator match query set to:

url.full: *

That will cause some of the issues of duplicates you're seeing.

Outside of removing duplicates, if you add the column of url.destination to your signals table what do you see? Are all those url's false positives, any true positives? Also what is the data type of url.destination vs. url.full?

You can see this information with this from dev tools like so, just replace xxx with the correct name of your source index:

GET xxxnsslog*/_mapping/field/url.destination

I'm interested if that is a keyword or if it is something else.

Double check url.full as well but locally that looks to should using the data type of keyword on my system.

GET filebeat-*/_mapping/field/url.full

# Should return something like this which shows it uses data type keyword
  "filebeat-8.0.0-2020.12.31-000005" : {
    "mappings" : {
      "url.full" : {
        "full_name" : "url.full",
        "mapping" : {
          "full" : {
            "type" : "keyword",
            "ignore_above" : 1024,
            "fields" : {
              "text" : {
                "type" : "text",
                "norms" : false
              }
            }
          }
        }
      }
    }
  },

The duplication only can be seen in discover, and not under indicator match detection rule threat match.

I already fixed the duplication on MISP side and whenever new feed is submitted or publish in MISP, by right it should only feed once. In my case, it didn't ingest the old ones.

I tested under dev tools for url.full: but too bad that I couldn't find exactly as what is shown in the discover index of filebeat whereby the url.full will be shown as url.full: (URL)

On file beat under discover, the url.full will shown as, url.full: (url) but on dev tools, it shown otherwise.
What is shown only the hashes and url with only legitimate domain such as VirusTotal.com. That is why I seen false positive results under threat match.

Is there any other way to get it match for only malicious url in my case?

=========
Results for: GET **nsslog/_mapping/field/url.destination

shown as:

{
"***nsslog" : {
"mappings" : {
"url.destination" : {
"full_name" : "url.destination",
"mapping" : {
"destination" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
}
}
}
}
}
}

========
Results for: GET filebeat-*/_mapping/field/url.full

Shown as:

{
"filebeat-7.10.1-2021.01.13-000001" : {
"mappings" : {
"url.full" : {
"full_name" : "url.full",
"mapping" : {
"full" : {
"type" : "keyword",
"ignore_above" : 1024,
"fields" : {
"text" : {
"type" : "text",
"norms" : false
}
}
}
}
}
}
}
}

One mapping has text as a default and the other has keyword as a default:

{
   "***nsslog":{
      "mappings":{
         "url.destination":{
            "full_name":"url.destination",
            "mapping":{
               "destination":{
                  "type":"text", <--- Uses text as default
                  "fields":{
                     "keyword":{
                        "type":"keyword",
                        "ignore_above":256
                     }
                  }
               }
            }
         }
      }
   }
}

vs the other way around with:

{
"filebeat-7.10.1-2021.01.13-000001" {
   "mappings":{
      "url.full":{
         "full_name":"url.full",
         "mapping":{
            "full":{
               "type":"keyword", <-- Uses keyword as default
               "ignore_above":1024,
               "fields":{
                  "text":{
                     "type":"text",
                     "norms":false
                  }
               }
            }
         }
      }
   }
}

You might be better off matching keyword against keyword by using something like this:

url.destination.keyword <--> url.full

Otherwise you might end up with something not expected as text will be analyzed and tokenized vs. keyword. Might still work with the way you're doing your mappings but I would suggest exact matches against keyword.

On file beat under discover, the url.full will shown as, url.full: (url) but on dev tools, it shown otherwise.

I don't know if I'm following this 100% but if you're seeing a lot of url.full coming up from dev tools that you do not want to try to match against then that would be the reason for all the false positives.

What you can do to eliminate them is to tune your KQL used in the Indicator index query to match only those things in your filebeat-* that are part of your indicator fields. You can use something like another exists in addition to your url.full: * if you need to filter down to your indicator lists and avoid any false positives from showing up.

1 Like

I created this and not getting any results. Is there a way we can test it out under Dev Tools and map to two indices just to test out?

====Rules Definition=====
Index patterns

***nsslog

Filters

action: Blocked

action: Allowed

Custom query

url.destination.keyword : *

Rule type

Indicator Match

Timeline template

Generic Network Timeline

Indicator index patterns

filebeat-*

Indicator mapping

url.destination.keyword MATCHES url.full

Indicator index query

url.full : *

========
Do take note, I also tried to exclude the filter of action allowed and blocked but seems also failing to receive alerts.

Hi, any recommendations on how to get this fixed?

It's tricky to help fix this without additional information.

I tested under dev tools for url.full: but too bad that I couldn't find exactly as what is shown in the discover index of filebeat whereby the url.full will be shown as url.full: (URL)

On file beat under discover, the url.full will shown as, url.full: (url) but on dev tools, it shown otherwise.
What is shown only the hashes and url with only legitimate domain such as VirusTotal.com. That is why I seen false positive results under threat match.

I'm not for sure I follow what is going on here. Are you seeing false positives and you know why you are seeing them and need help filtering those out?

If you're not seeing something immediately showing up it could be that you have correctly setup everything but the matching of the indicators are taking a very long time and could be timing out if your list is very large.

  • What is the count of your list? Is it 100s, 1k's, 1 million?
  • What is the count of your events every 5 minutes? Is it 100s, 1,000's, millions?
  • Can we a query the list such as the top 10?
  • Can we get an export of your rule in question?
  • Can we get an export of your target index mapping?
  • Can we get a sample of your target index data?

The more information we can have, the better we can help out.

Will get back to you on these request soon. Thanks Frank