Indicator Match Detection Rule Not Matched and Mapped to Intel Feeds

[Frank_Hassanabad]

For duplications, there have been other posts depending on which version of the misp module you have installed which should help you with removing duplicates:

MISP and Elastic Security

I have a use case where I want to index attributes from MISP to Elasticsearch for using the new Threat Matching rule in Elastic Security 7.10. Everything is working fine but the issue is when I try to ingest more attributes from misp along with the new one it also again ingests the old ones creating huge duplicates so I delete the whole index before ingesting attributes from MISP. Is there a way where I can prevent the duplicate events to be ingested again?

When it queries each item from the list with the indicator match query set to:

url.full: *

That will cause some of the issues of duplicates you're seeing.

Outside of removing duplicates, if you add the column of url.destination to your signals table what do you see? Are all those url's false positives, any true positives? Also what is the data type of url.destination vs. url.full?

You can see this information with this from dev tools like so, just replace xxx with the correct name of your source index:

GET xxxnsslog*/_mapping/field/url.destination

I'm interested if that is a keyword or if it is something else.

Double check url.full as well but locally that looks to should using the data type of keyword on my system.

GET filebeat-*/_mapping/field/url.full

# Should return something like this which shows it uses data type keyword
  "filebeat-8.0.0-2020.12.31-000005" : {
    "mappings" : {
      "url.full" : {
        "full_name" : "url.full",
        "mapping" : {
          "full" : {
            "type" : "keyword",
            "ignore_above" : 1024,
            "fields" : {
              "text" : {
                "type" : "text",
                "norms" : false
              }
            }
          }
        }
      }
    }
  },