Filebeat Threat Intel Module Errors

tofubeats · November 17, 2022, 4:59pm

Hi,

I am setting up MISP servers and Threat Intel Module. I can get the threat intel module to bring in IOCs from other feeds, but MISP is creating issues.

I keep getting the following error.message: "cannot access method/field [size] from a null def reference"

I am making sure there are infact new events to ingest by manually creating new ones just to see them pop up in Elastic. I can see in the MISP UI that the Auth key I gave Filebeat is being used. Connectivity does not seem to be an issue either, I can ping both machines both ways. I'm not sure where else to look to trouble shoot and would appreciate any guidance.

I have not changed the default pipeline after loading it. This is what the processors look like:

    [
      {
        "set": {
          "value": "{{_ingest.timestamp}}",
          "field": "event.ingested"
        }
      },
      {
        "set": {
          "field": "event.kind",
          "value": "enrichment"
        }
      },
      {
        "set": {
          "field": "event.category",
          "value": "threat"
        }
      },
      {
        "set": {
          "field": "event.type",
          "value": "indicator"
        }
      },
      {
        "rename": {
          "field": "json.Event",
          "target_field": "threatintel.misp",
          "ignore_missing": true
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.provider",
          "value": "misp",
          "if": "ctx?.threatintel?.misp?.Orgc?.local != 'false'"
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.provider",
          "value": "{{misp.Orgc.name}}",
          "if": "ctx?.threatintel?.misp?.Orgc?.local == 'false'",
          "ignore_empty_value": true
        }
      },
      {
        "remove": {
          "field": [
            "threatintel.misp.ShadowAttribute",
            "threatintel.misp.RelatedEvent",
            "threatintel.misp.Galaxy",
            "threatintel.misp.Attribute.Galaxy",
            "threatintel.misp.Attribute.ShadowAttribute",
            "threatintel.misp.EventReport",
            "threatintel.misp.Object.Attribute.Galaxy",
            "threatintel.misp.Object.Attribute.ShadowAttribute",
            "message"
          ],
          "ignore_missing": true
        }
      },
      {
        "remove": {
          "if": "ctx?.threatintel?.misp?.Attribute.size() == 0",
          "field": [
            "threatintel.misp.Attribute"
          ],
          "ignore_missing": true
        }
      },
      {
        "remove": {
          "field": [
            "threatintel.misp.Object"
          ],
          "ignore_missing": true,
          "if": "ctx?.threatintel?.misp?.Object.size() == 0"
        }
      },
      {
        "date": {
          "field": "threatintel.misp.timestamp",
          "formats": [
            "UNIX"
          ],
          "ignore_failure": true
        }
      },
      {
        "rename": {
          "ignore_missing": true,
          "field": "threatintel.misp.Attribute",
          "target_field": "threatintel.misp.attribute"
        }
      },
      {
        "rename": {
          "ignore_missing": true,
          "field": "threatintel.misp.Object",
          "target_field": "threatintel.misp.object"
        }
      },
      {
        "rename": {
          "ignore_missing": true,
          "field": "threatintel.misp.object.Attribute",
          "target_field": "threatintel.misp.object.attribute"
        }
      },
      {
        "rename": {
          "target_field": "threatintel.misp.orgc",
          "ignore_missing": true,
          "field": "threatintel.misp.Orgc"
        }
      },
      {
        "rename": {
          "field": "threatintel.misp.Org",
          "target_field": "threatintel.misp.org",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "threatintel.misp.Tag",
          "target_field": "threatintel.misp.tag",
          "ignore_missing": true
        }
      },
      {
        "rename": {
          "field": "threatintel.misp.attribute",
          "target_field": "threatintel.misp.context.attribute",
          "ignore_missing": true,
          "if": "ctx?.threatintel?.misp?.object != null"
        }
      },
      {
        "rename": {
          "field": "threatintel.misp.object.attribute",
          "target_field": "threatintel.misp.attribute",
          "ignore_missing": true,
          "if": "ctx?.threatintel?.misp?.object != null"
        }
      },
      {
        "rename": {
          "target_field": "threatintel.indicator.first_seen",
          "ignore_missing": true,
          "field": "threatintel.misp.attribute.first_seen"
        }
      },
      {
        "rename": {
          "field": "threatintel.misp.attribute.last_seen",
          "target_field": "threatintel.indicator.last_seen",
          "ignore_missing": true
        }
      },
      {
        "convert": {
          "field": "threatintel.misp.analysis",
          "type": "long",
          "target_field": "threatintel.indicator.scanner_stats",
          "ignore_missing": true
        }
      },
      {
        "convert": {
          "type": "long",
          "ignore_missing": true,
          "field": "threatintel.misp.threat_level_id"
        }
      },
      {
        "set": {
          "value": "file",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && (['md5', 'impfuzzy', 'imphash', 'pehash', 'sha1', 'sha224', 'sha256', 'sha3-224', 'sha3-256', 'sha3-384', 'sha3-512', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'tlsh', 'vhash'].contains(ctx.threatintel?.misp?.attribute?.type) || ctx.threatintel?.misp?.attribute?.type.startsWith('filename'))",
          "field": "threatintel.indicator.type"
        }
      },
      {
        "rename": {
          "ignore_missing": true,
          "if": "ctx?.threatintel?.indicator?.type == 'file' && ctx?.threatintel?.misp?.attribute?.type != null && !ctx?.threatintel?.misp?.attribute?.type.startsWith('filename')",
          "field": "threatintel.misp.attribute.value",
          "target_field": "threatintel.indicator.file.hash.{{threatintel.misp.attribute.type}}"
        }
      },
      {
        "rename": {
          "ignore_missing": true,
          "if": "ctx?.threatintel?.indicator?.type == 'file' && ctx?.threatintel?.misp?.attribute?.type == 'filename'",
          "field": "threatintel.misp.attribute.value",
          "target_field": "threatintel.indicator.file.name"
        }
      },
      {
        "grok": {
          "ignore_missing": true,
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|')",
          "field": "threatintel.misp.attribute.type",
          "patterns": [
            "%{WORD}\\|%{WORD:_tmp.hashtype}"
          ]
        }
      },
      {
        "grok": {
          "patterns": [
            "%{DATA:threatintel.indicator.file.name}\\|%{GREEDYDATA:_tmp.hashvalue}"
          ],
          "ignore_missing": true,
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|')",
          "field": "threatintel.misp.attribute.value"
        }
      },
      {
        "set": {
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|') && ctx?._tmp?.hashvalue != null && ctx?._tmp?.hashtype != null",
          "field": "threatintel.indicator.file.hash.{{_tmp.hashtype}}",
          "value": "{{_tmp.hashvalue}}"
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.type",
          "value": "url",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['url', 'link', 'uri'].contains(ctx?.threatintel?.misp?.attribute?.type)"
        }
      },
      {
        "uri_parts": {
          "remove_if_successful": true,
          "if": "ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.misp?.attribute?.type != 'uri'",
          "field": "threatintel.misp.attribute.value",
          "target_field": "threatintel.indicator.url",
          "keep_original": true
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.url.full",
          "value": "{{{threatintel.indicator.url.original}}}",
          "ignore_empty_value": true,
          "if": "ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.misp?.attribute?.type != 'uri'"
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.type",
          "value": "windows-registry-key",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('regkey')"
        }
      },
      {
        "rename": {
          "if": "ctx?.threatintel?.indicator?.type == 'windows-registry-key' && ctx?.threatintel?.misp?.attribute?.type == 'regkey'",
          "field": "threatintel.misp.attribute.value",
          "target_field": "threatintel.indicator.registry.key",
          "ignore_missing": true
        }
      },
      {
        "grok": {
          "field": "threatintel.misp.attribute.value",
          "patterns": [
            "%{DATA:threatintel.indicator.registry.key}\\|%{DATA:threatintel.indicator.registry.value}"
          ],
          "ignore_missing": true,
          "if": "ctx?.threatintel?.misp?.attribute?.type == 'regkey|value'"
        }
      },
      {
        "set": {
          "value": "autonomous-system",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type == 'AS'",
          "field": "threatintel.indicator.type"
        }
      },
      {
        "convert": {
          "if": "ctx?.threatintel?.indicator?.type == 'autonomous-system'",
          "field": "threatintel.misp.attribute.value",
          "type": "long",
          "target_field": "threatintel.indicator.as.number",
          "ignore_missing": true
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.type",
          "value": "domain-name",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && (ctx?.threatintel?.misp?.attribute?.type == 'hostname' || ctx?.threatintel?.misp?.attribute?.type.startsWith('domain'))"
        }
      },
      {
        "set": {
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['ip-src', 'ip-src|port', 'ip-dst', 'ip-dst|port'].contains(ctx?.threatintel?.misp?.attribute?.type)",
          "field": "threatintel.indicator.type",
          "value": "ipv4-addr"
        }
      },
      {
        "rename": {
          "target_field": "threatintel.indicator.url.domain",
          "ignore_missing": true,
          "if": "ctx?.threatintel?.indicator?.type == 'domain-name' && ctx?.threatintel?.misp?.attribute?.type != 'domain|ip' && ctx.threatintel?.indicator?.url?.domain == null",
          "field": "threatintel.misp.attribute.value"
        }
      },
      {
        "rename": {
          "target_field": "threatintel.indicator.ip",
          "ignore_missing": true,
          "if": "ctx?.threatintel?.indicator?.type == 'ipv4-addr' && ctx?.threatintel?.misp?.attribute?.type != 'domain|ip' && !['ip-src|port', 'ip-dst|port'].contains(ctx?.threatintel?.misp?.attribute?.type)",
          "field": "threatintel.misp.attribute.value"
        }
      },
      {
        "grok": {
          "patterns": [
            "%{DATA:threatintel.indicator.url.domain}\\|%{IP:threatintel.indicator.ip}"
          ],
          "ignore_missing": true,
          "if": "ctx.threatintel?.misp?.attribute?.type == 'domain|ip' && ctx.threatintel?.indicator?.url?.domain == null",
          "field": "threatintel.misp.attribute.value"
        }
      },
      {
        "grok": {
          "if": "['ip-src|port', 'ip-dst|port'].contains(ctx.threatintel?.misp?.attribute?.type)",
          "field": "threatintel.misp.attribute.value",
          "patterns": [
            "%{IP:threatintel.indicator.ip}\\|%{NUMBER:threatintel.indicator.port}"
          ],
          "ignore_missing": true
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.type",
          "value": "email-addr",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['email-dst', 'email-src'].contains(ctx.threatintel?.misp?.attribute?.type)"
        }
      },
      {
        "set": {
          "value": "email-message",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx.threatintel?.misp?.attribute?.type.startsWith('email') && !['email-dst', 'email-src'].contains(ctx.threatintel?.misp?.attribute?.type)",
          "field": "threatintel.indicator.type"
        }
      },
      {
        "rename": {
          "field": "threatintel.misp.attribute.value",
          "target_field": "threatintel.indicator.email.address",
          "ignore_missing": true,
          "if": "ctx?.threatintel?.indicator?.type == 'email-addr'"
        }
      },
      {
        "rename": {
          "field": "threatintel.misp.event_creator_email",
          "target_field": "user.email",
          "ignore_missing": true
        }
      },
      {
        "append": {
          "field": "user.roles",
          "value": "reporting_user",
          "if": "ctx?.user?.email != null"
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.type",
          "value": "mac-addr",
          "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['mac-address', 'mac-eui-64'].contains(ctx.threatintel?.misp?.attribute?.type)"
        }
      },
      {
        "rename": {
          "target_field": "threatintel.indicator.mac",
          "ignore_missing": true,
          "if": "ctx?.threatintel?.indicator?.type == 'mac-addr'",
          "field": "threatintel.misp.attribute.value"
        }
      },
      {
        "script": {
          "lang": "painless",
          "if": "ctx?.threatintel?.misp?.tag != null",
          "source": "def tags = ctx.threatintel.misp.tag.stream()\n   .map(t -> t.name.replace('\\\\', '').replace('\"', ''))\n   .collect(Collectors.toList());\ndef tlpTags = tags.stream()\n   .filter(t -> t.startsWith('tlp:'))\n   .map(t -> t.replace('tlp:', ''))\n   .collect(Collectors.toList());\n\nctx.tags = tags;\nctx.threatintel.indicator.marking = [ 'tlp': tlpTags ];\n"
        }
      },
      {
        "set": {
          "field": "threatintel.indicator.type",
          "value": "unknown",
          "if": "ctx?.threatintel?.indicator?.type == null"
        }
      },
      {
        "script": {
          "lang": "painless",
          "if": "ctx?.threatintel != null",
          "source": "void handleMap(Map map) {\n  for (def x : map.values()) {\n    if (x instanceof Map) {\n        handleMap(x);\n    } else if (x instanceof List) {\n        handleList(x);\n    }\n  }\nmap.values().removeIf(v -> v == null);\n}\nvoid handleList(List list) {\n  for (def x : list) {\n      if (x instanceof Map) {\n          handleMap(x);\n      } else if (x instanceof List) {\n          handleList(x);\n      }\n  }\n}\nhandleMap(ctx);\n"
        }
      },
      {
        "remove": {
          "field": [
            "threatintel.misp.attribute.value"
          ],
          "ignore_missing": true,
          "if": "ctx?.threatintel?.indicator?.type != 'unknown'"
        }
      },
      {
        "remove": {
          "field": [
            "threatintel.misp.object"
          ],
          "ignore_missing": true
        }
      },
      {
        "remove": {
          "field": [
            "threatintel.misp.Attribute.timestamp",
            "threatintel.misp.timestamp",
            "threatintel.misp.tag",
            "threatintel.misp.org",
            "threatintel.misp.analysis",
            "_tmp",
            "json"
          ],
          "ignore_missing": true
        }
      }
    ]

Failure processors

    [
      {
        "set": {
          "field": "error.message",
          "value": "{{ _ingest.on_failure_message }}"
        }
      }
    ]

[
  {
    "set": {
      "value": "{{_ingest.timestamp}}",
      "field": "event.ingested"
    }
  },
  {
    "set": {
      "field": "event.kind",
      "value": "enrichment"
    }
  },
  {
    "set": {
      "field": "event.category",
      "value": "threat"
    }
  },
  {
    "set": {
      "field": "event.type",
      "value": "indicator"
    }
  },
  {
    "rename": {
      "field": "json.Event",
      "target_field": "threatintel.misp",
      "ignore_missing": true
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.provider",
      "value": "misp",
      "if": "ctx?.threatintel?.misp?.Orgc?.local != 'false'"
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.provider",
      "value": "{{misp.Orgc.name}}",
      "if": "ctx?.threatintel?.misp?.Orgc?.local == 'false'",
      "ignore_empty_value": true
    }
  },
  {
    "remove": {
      "field": [
        "threatintel.misp.ShadowAttribute",
        "threatintel.misp.RelatedEvent",
        "threatintel.misp.Galaxy",
        "threatintel.misp.Attribute.Galaxy",
        "threatintel.misp.Attribute.ShadowAttribute",
        "threatintel.misp.EventReport",
        "threatintel.misp.Object.Attribute.Galaxy",
        "threatintel.misp.Object.Attribute.ShadowAttribute",
        "message"
      ],
      "ignore_missing": true
    }
  },
  {
    "remove": {
      "if": "ctx?.threatintel?.misp?.Attribute.size() == 0",
      "field": [
        "threatintel.misp.Attribute"
      ],
      "ignore_missing": true
    }
  },
  {
    "remove": {
      "field": [
        "threatintel.misp.Object"
      ],
      "ignore_missing": true,
      "if": "ctx?.threatintel?.misp?.Object.size() == 0"
    }
  },
  {
    "date": {
      "field": "threatintel.misp.timestamp",
      "formats": [
        "UNIX"
      ],
      "ignore_failure": true
    }
  },
  {
    "rename": {
      "ignore_missing": true,
      "field": "threatintel.misp.Attribute",
      "target_field": "threatintel.misp.attribute"
    }
  },
  {
    "rename": {
      "ignore_missing": true,
      "field": "threatintel.misp.Object",
      "target_field": "threatintel.misp.object"
    }
  },
  {
    "rename": {
      "ignore_missing": true,
      "field": "threatintel.misp.object.Attribute",
      "target_field": "threatintel.misp.object.attribute"
    }
  },
  {
    "rename": {
      "target_field": "threatintel.misp.orgc",
      "ignore_missing": true,
      "field": "threatintel.misp.Orgc"
    }
  },
  {
    "rename": {
      "field": "threatintel.misp.Org",
      "target_field": "threatintel.misp.org",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "threatintel.misp.Tag",
      "target_field": "threatintel.misp.tag",
      "ignore_missing": true
    }
  },
  {
    "rename": {
      "field": "threatintel.misp.attribute",
      "target_field": "threatintel.misp.context.attribute",
      "ignore_missing": true,
      "if": "ctx?.threatintel?.misp?.object != null"
    }
  },
  {
    "rename": {
      "field": "threatintel.misp.object.attribute",
      "target_field": "threatintel.misp.attribute",
      "ignore_missing": true,
      "if": "ctx?.threatintel?.misp?.object != null"
    }
  },
  {
    "rename": {
      "target_field": "threatintel.indicator.first_seen",
      "ignore_missing": true,
      "field": "threatintel.misp.attribute.first_seen"
    }
  },
  {
    "rename": {
      "field": "threatintel.misp.attribute.last_seen",
      "target_field": "threatintel.indicator.last_seen",
      "ignore_missing": true
    }
  },
  {
    "convert": {
      "field": "threatintel.misp.analysis",
      "type": "long",
      "target_field": "threatintel.indicator.scanner_stats",
      "ignore_missing": true
    }
  },
  {
    "convert": {
      "type": "long",
      "ignore_missing": true,
      "field": "threatintel.misp.threat_level_id"
    }
  },
  {
    "set": {
      "value": "file",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && (['md5', 'impfuzzy', 'imphash', 'pehash', 'sha1', 'sha224', 'sha256', 'sha3-224', 'sha3-256', 'sha3-384', 'sha3-512', 'sha384', 'sha512', 'sha512/224', 'sha512/256', 'ssdeep', 'tlsh', 'vhash'].contains(ctx.threatintel?.misp?.attribute?.type) || ctx.threatintel?.misp?.attribute?.type.startsWith('filename'))",
      "field": "threatintel.indicator.type"
    }
  },
  {
    "rename": {
      "ignore_missing": true,
      "if": "ctx?.threatintel?.indicator?.type == 'file' && ctx?.threatintel?.misp?.attribute?.type != null && !ctx?.threatintel?.misp?.attribute?.type.startsWith('filename')",
      "field": "threatintel.misp.attribute.value",
      "target_field": "threatintel.indicator.file.hash.{{threatintel.misp.attribute.type}}"
    }
  },
  {
    "rename": {
      "ignore_missing": true,
      "if": "ctx?.threatintel?.indicator?.type == 'file' && ctx?.threatintel?.misp?.attribute?.type == 'filename'",
      "field": "threatintel.misp.attribute.value",
      "target_field": "threatintel.indicator.file.name"
    }
  },
  {
    "grok": {
      "ignore_missing": true,
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|')",
      "field": "threatintel.misp.attribute.type",
      "patterns": [
        "%{WORD}\\|%{WORD:_tmp.hashtype}"
      ]
    }
  },
  {
    "grok": {
      "patterns": [
        "%{DATA:threatintel.indicator.file.name}\\|%{GREEDYDATA:_tmp.hashvalue}"
      ],
      "ignore_missing": true,
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|')",
      "field": "threatintel.misp.attribute.value"
    }
  },
  {
    "set": {
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('filename|') && ctx?._tmp?.hashvalue != null && ctx?._tmp?.hashtype != null",
      "field": "threatintel.indicator.file.hash.{{_tmp.hashtype}}",
      "value": "{{_tmp.hashvalue}}"
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.type",
      "value": "url",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['url', 'link', 'uri'].contains(ctx?.threatintel?.misp?.attribute?.type)"
    }
  },
  {
    "uri_parts": {
      "remove_if_successful": true,
      "if": "ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.misp?.attribute?.type != 'uri'",
      "field": "threatintel.misp.attribute.value",
      "target_field": "threatintel.indicator.url",
      "keep_original": true
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.url.full",
      "value": "{{{threatintel.indicator.url.original}}}",
      "ignore_empty_value": true,
      "if": "ctx?.threatintel?.indicator?.type == 'url' && ctx?.threatintel?.misp?.attribute?.type != 'uri'"
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.type",
      "value": "windows-registry-key",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type.startsWith('regkey')"
    }
  },
  {
    "rename": {
      "if": "ctx?.threatintel?.indicator?.type == 'windows-registry-key' && ctx?.threatintel?.misp?.attribute?.type == 'regkey'",
      "field": "threatintel.misp.attribute.value",
      "target_field": "threatintel.indicator.registry.key",
      "ignore_missing": true
    }
  },
  {
    "grok": {
      "field": "threatintel.misp.attribute.value",
      "patterns": [
        "%{DATA:threatintel.indicator.registry.key}\\|%{DATA:threatintel.indicator.registry.value}"
      ],
      "ignore_missing": true,
      "if": "ctx?.threatintel?.misp?.attribute?.type == 'regkey|value'"
    }
  },
  {
    "set": {
      "value": "autonomous-system",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx?.threatintel?.misp?.attribute?.type == 'AS'",
      "field": "threatintel.indicator.type"
    }
  },
  {
    "convert": {
      "if": "ctx?.threatintel?.indicator?.type == 'autonomous-system'",
      "field": "threatintel.misp.attribute.value",
      "type": "long",
      "target_field": "threatintel.indicator.as.number",
      "ignore_missing": true
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.type",
      "value": "domain-name",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && (ctx?.threatintel?.misp?.attribute?.type == 'hostname' || ctx?.threatintel?.misp?.attribute?.type.startsWith('domain'))"
    }
  },
  {
    "set": {
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['ip-src', 'ip-src|port', 'ip-dst', 'ip-dst|port'].contains(ctx?.threatintel?.misp?.attribute?.type)",
      "field": "threatintel.indicator.type",
      "value": "ipv4-addr"
    }
  },
  {
    "rename": {
      "target_field": "threatintel.indicator.url.domain",
      "ignore_missing": true,
      "if": "ctx?.threatintel?.indicator?.type == 'domain-name' && ctx?.threatintel?.misp?.attribute?.type != 'domain|ip' && ctx.threatintel?.indicator?.url?.domain == null",
      "field": "threatintel.misp.attribute.value"
    }
  },
  {
    "rename": {
      "target_field": "threatintel.indicator.ip",
      "ignore_missing": true,
      "if": "ctx?.threatintel?.indicator?.type == 'ipv4-addr' && ctx?.threatintel?.misp?.attribute?.type != 'domain|ip' && !['ip-src|port', 'ip-dst|port'].contains(ctx?.threatintel?.misp?.attribute?.type)",
      "field": "threatintel.misp.attribute.value"
    }
  },
  {
    "grok": {
      "patterns": [
        "%{DATA:threatintel.indicator.url.domain}\\|%{IP:threatintel.indicator.ip}"
      ],
      "ignore_missing": true,
      "if": "ctx.threatintel?.misp?.attribute?.type == 'domain|ip' && ctx.threatintel?.indicator?.url?.domain == null",
      "field": "threatintel.misp.attribute.value"
    }
  },
  {
    "grok": {
      "if": "['ip-src|port', 'ip-dst|port'].contains(ctx.threatintel?.misp?.attribute?.type)",
      "field": "threatintel.misp.attribute.value",
      "patterns": [
        "%{IP:threatintel.indicator.ip}\\|%{NUMBER:threatintel.indicator.port}"
      ],
      "ignore_missing": true
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.type",
      "value": "email-addr",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['email-dst', 'email-src'].contains(ctx.threatintel?.misp?.attribute?.type)"
    }
  },
  {
    "set": {
      "value": "email-message",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ctx.threatintel?.misp?.attribute?.type.startsWith('email') && !['email-dst', 'email-src'].contains(ctx.threatintel?.misp?.attribute?.type)",
      "field": "threatintel.indicator.type"
    }
  },
  {
    "rename": {
      "field": "threatintel.misp.attribute.value",
      "target_field": "threatintel.indicator.email.address",
      "ignore_missing": true,
      "if": "ctx?.threatintel?.indicator?.type == 'email-addr'"
    }
  },
  {
    "rename": {
      "field": "threatintel.misp.event_creator_email",
      "target_field": "user.email",
      "ignore_missing": true
    }
  },
  {
    "append": {
      "field": "user.roles",
      "value": "reporting_user",
      "if": "ctx?.user?.email != null"
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.type",
      "value": "mac-addr",
      "if": "ctx?.threatintel?.misp?.attribute?.type != null && ['mac-address', 'mac-eui-64'].contains(ctx.threatintel?.misp?.attribute?.type)"
    }
  },
  {
    "rename": {
      "target_field": "threatintel.indicator.mac",
      "ignore_missing": true,
      "if": "ctx?.threatintel?.indicator?.type == 'mac-addr'",
      "field": "threatintel.misp.attribute.value"
    }
  },
  {
    "script": {
      "lang": "painless",
      "if": "ctx?.threatintel?.misp?.tag != null",
      "source": "def tags = ctx.threatintel.misp.tag.stream()\n   .map(t -> t.name.replace('\\\\', '').replace('\"', ''))\n   .collect(Collectors.toList());\ndef tlpTags = tags.stream()\n   .filter(t -> t.startsWith('tlp:'))\n   .map(t -> t.replace('tlp:', ''))\n   .collect(Collectors.toList());\n\nctx.tags = tags;\nctx.threatintel.indicator.marking = [ 'tlp': tlpTags ];\n"
    }
  },
  {
    "set": {
      "field": "threatintel.indicator.type",
      "value": "unknown",
      "if": "ctx?.threatintel?.indicator?.type == null"
    }
  },
  {
    "script": {
      "lang": "painless",
      "if": "ctx?.threatintel != null",
      "source": "void handleMap(Map map) {\n  for (def x : map.values()) {\n    if (x instanceof Map) {\n        handleMap(x);\n    } else if (x instanceof List) {\n        handleList(x);\n    }\n  }\nmap.values().removeIf(v -> v == null);\n}\nvoid handleList(List list) {\n  for (def x : list) {\n      if (x instanceof Map) {\n          handleMap(x);\n      } else if (x instanceof List) {\n          handleList(x);\n      }\n  }\n}\nhandleMap(ctx);\n"
    }
  },
  {
    "remove": {
      "field": [
        "threatintel.misp.attribute.value"
      ],
      "ignore_missing": true,
      "if": "ctx?.threatintel?.indicator?.type != 'unknown'"
    }
  },
  {
    "remove": {
      "field": [
        "threatintel.misp.object"
      ],
      "ignore_missing": true
    }
  },
  {
    "remove": {
      "field": [
        "threatintel.misp.Attribute.timestamp",
        "threatintel.misp.timestamp",
        "threatintel.misp.tag",
        "threatintel.misp.org",
        "threatintel.misp.analysis",
        "_tmp",
        "json"
      ],
      "ignore_missing": true
    }
  }
]

animoL · November 18, 2022, 8:54am

We have the same identical issue. Everything is correctly configured and we can even ingest data from the MISP to any other platform.
However, Elasticsearch with the MISP integration seems to throw the same exact error without importing any data.

animoL · November 19, 2022, 9:33am

We managed to dig some more and apparently there's an issue with the processor imported by the integration that can't correctly parse what MISP sends:

{"type":"mapper_parsing_exception","reason":"failed to parse field [event.original] of type [keyword] in document with id 'LFJOm7Chf3RntOaij7qosHw4/98='

illegal_argument_exception","reason":"Document contains at least one immense term in field=\"event.original\" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped. Please correct the analyzer to not produce such terms. The prefix of the first immense term is: '[123, 34, 69, 118, 101, 110, 116, 34, 58, 123, 34, 65, 116, 116, 114, 105, 98, 117, 116, 101, 34, 58, 123, 34, 71, 97, 108, 97, 120, 121]...'"}}, dropping event!

system · December 17, 2022, 11:34am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat 7.17, error "cannot access method/field [size] from a null def reference" while trying to download IOCs from MISP Beats filebeat	0	175	May 10, 2024
Bug: No Misp event data send to Kibana when Threat intel module used Beats beats-module , filebeat	6	967	April 19, 2023
[BUG] Threatintel MISP Plugin runs in endless loop Beats beats-module , filebeat	2	419	December 4, 2023
Msip threat intel import not working Elastic Security	3	386	October 3, 2021
Filebeat ThreatIntel MISP Module Beats filebeat	3	669	June 25, 2021

Filebeat Threat Intel Module Errors

Related topics