MISP + Alerts

[VellayLoket]

I had connect MISP to ELK with filebeat.
So now i have index named filebeat, there are many IOCs.
Next i have index with network activity from workstations.
So i want to match IP from winlog index with IOCs from MISP index.
Here how the rule looks like:

So this rule finished with error with timeout, how to fix it???
"An error occurred during rule execution: message: "Request timed out" name: "Malicious network activity (MISP)" id: "66449ca0-fac4-11ed-8785-31b707a79fa9" rule id: "b8b8d859-dda9-4d42-9b30-f0ef56b7d4ba" signals index: ".siem-signals-default""

[RylandHerrick]

Hi @VellayLoket! Thanks for checking out the Indicator Match rule type.

A general rule timeout could mean a number of things, but most likely the rule is timing out due to it having more work than it can perform in the given time period. There are many ways to address this, but they typically involve paring down that work:

1. Reduce the scope of the IOCs
   • our prebuilt IM rule only includes IOCs from the last 30 days, for example, and we often recommend including existence filters for the fields upon which you're matching, but reducing the set of IOCs as much as possible will always improve performance.
2. Reduce the scope of source documents
   • much of the same logic as above applies here: in your case, a winlog.event_data.DestinationIp: * couldn't hurt, but scoping that even further to a particular IP range would be even better.
3. Reduce the time range of source documents
   • it's not shown in your screenshot, but increasing both the interval/lookback so that the rule runs less often (but on more data) could help in a situation where the number of IOCs is the limiting factor.

One more note: it looks like you may be on 7.x (judging by the data you've provided); it's worth noting that the Indicator Match rule has both performance improvements and better error output in the 8.x series.

[VellayLoket]

@RylandHerrick

One more note: it looks like you may be on 7.x (judging by the data you've provided); it's worth noting that the Indicator Match rule has both performance improvements and better error output in the 8.x series.

We're use 7.15 version.
Btw, i had try to optimize like you write here, here the screenshots of settings:

So i check only external IPs, filter winlogbeat data with event code = 3 (only network events) and get event from last 5 minutes (@timestamp >= now-5m), with MISP indicators i get only last 30 days and only indicators with IP IOCs

and with all this optimisations nothing changed - the same error - timeout

[RylandHerrick]

Sorry to hear that you're still having trouble. I assume you've seen our support guidelines, but it's worth mentioning just in case.

Can you perform some queries to get a sense for the volume of data your rule is working with?

• How many IOCs exist in the last 30 days? How many are ip-dist?
• How many network events are ingested in a 5 minute window?

A few other bases to cover:

• You're adding time clauses to your queries, but there's also an "outer" range for rule execution, as defined by the Schedule: "Rule runs every" and "Additional look-back time" fields. Could you please share that configuration from your rule? A full NDJSON export of the rule would be ideal.
• Is all of your data on the hot tier?
• Is it possible for there to be ip-dist IOCs without a destination.ip? That may be another optimization.
• Is there duplication of your IOC data? The IM rule does not deduplicate identical values, so duplicates will waste CPU.

In terms of troubleshooting: have you seen the rule complete successfully? I would try getting a minimal use case to work: e.g. inspect your data, find a "match," and update the rule to query exactly those two documents (via _id, say). That will help to eliminate a whole class of issues here.

[VellayLoket]

• How many IOCs exist in the last 30 days? How many are ip-dist?
  69,014 for 30 days. This is only which have ip-dist.

• How many network events are ingested in a 5 minute window?
  2620

• You're adding time clauses to your queries, but there's also an "outer" range for rule execution, as defined by the Schedule: "Rule runs every" and "Additional look-back time" fields. Could you please share that configuration from your rule? A full NDJSON export of the rule would be ideal.
  I play a little with configuration. Now it looks like this:

{"id":"66449ca0-fac4-11ed-8785-31b707a79fa9","updated_at":"2023-05-28T23:22:01.513Z","updated_by":"elastic","created_at":"2023-05-25T06:21:36.657Z","created_by":"elastic","name":"Malicious network activity (MISP)","tags":["MISP"],"interval":"5m","enabled":true,"description":"MISP malicious IP feeds","risk_score":73,"severity":"high","license":"","output_index":".siem-signals-default","meta":{"from":"1m","kibana_siem_app_url":"https://<redacted>:5601/app/security"},"author":[],"false_positives":[],"from":"now-360s","rule_id":"b8b8d859-dda9-4d42-9b30-f0ef56b7d4ba","max_signals":100,"risk_score_mapping":[],"severity_mapping":[],"threat":[],"to":"now","references":[],"version":30,"exceptions_list":[],"immutable":false,"type":"threat_match","language":"kuery","index":["winlogbeat-*"],"query":"not winlog.event_data.DestinationIp : 10.* and not winlog.event_data.DestinationIp : 255.255.255.255","filters":[{"$state":{"store":"appState"},"meta":{"negate":false,"alias":null,"index":"185fa530-d435-11ed-9b91-b596ef7bb041","disabled":false,"params":{"query":"Microsoft-Windows-Sysmon"},"type":"phrase","key":"event.provider"},"query":{"match_phrase":{"event.provider":"Microsoft-Windows-Sysmon"}}},{"$state":{"store":"appState"},"meta":{"negate":false,"alias":null,"index":"185fa530-d435-11ed-9b91-b596ef7bb041","disabled":false,"type":"phrase","params":{"query":"3"},"key":"event.code"},"query":{"match_phrase":{"event.code":"3"}}}],"saved_id":"Sysmon Network activity (last 15 minutes)","threat_filters":[],"threat_query":"@timestamp >= now-6h and misp.threat_indicator.type: \"ip-dst\" ","threat_mapping":[{"entries":[{"field":"winlog.event_data.DestinationIp.keyword","type":"mapping","value":"destination.ip"}]}],"threat_language":"kuery","threat_index":["filebeat-*"],"threat_indicator_path":"","throttle":"no_actions","actions":[]}
{"exported_count":1,"missing_rules":[],"missing_rules_count":0}

• Is all of your data on the hot tier?
  Yes, i don't use any other types of tires.

• Is it possible for there to be ip-dist IOCs without a destination.ip? That may be another optimization.
  It's impossible. If the event have type = ip-dst it always have the field destination.ip with data.

• Is there duplication of your IOC data? The IM rule does not deduplicate identical values, so duplicates will waste CPU.
  Really, i don't know. Its data from MISP.

[VellayLoket]

So while i play with setting i got that with this setting the rule starts to work well:

@timestamp >= now-6h and misp.threat_indicator.type: "ip-dst"

[VellayLoket]

I think i can optimize the rule with filter local ip ranges.
Can i filter 172.16 - 172.31 with one expression (smth like not destination.ip=10.* but for that range)?

[RylandHerrick]

If your data is mapped as ip you should be able to use a range query to filter your data.

So while i play with setting i got that with this setting the rule starts to work well:

So this change seems to imply that the number of IOCs is your issue; would you be able to validate that?

• How many IOCs are in the last 6h? In the last 24h? In the last 7d?
• If you increase the IOC range in your rule, (i.e. 24h, 2d, 7d), at what point does the rule start to time out?