[BUG] Threatintel MISP Plugin runs in endless loop

[hti]

Hello,
we are running filebeat 8.8.1 and use the threatintel module to ingest data from MISP to elasticsearch.
While it is running well most of the time, some MISP events (probably those with many attributes) will result in an endless loop of the filebeat module. The filebeat instance creates a huge number (possible infinite) of entries in elasticsearch. The behaviour is the same if a file output is used in filebeat.

We narrowed down the cause by modifying threatintel/misp/config/config.yml to

response.split:
  target: body.response
#  split:
#    target: body.Event.Attribute
#    ignore_empty_value: true
#    keep_parent: true
#    split:
#      target: body.Event.Object
#      keep_parent: true
#      split:
#        target: body.Event.Object.Attribute
#        keep_parent: true
response.request_body_on_pagination: true
response.pagination:
- set:
    target: body.page
    value: '[[if (ne (len .last_response.body.response) 0)]][[add .last_response.page 1]][[end]]'
    fail_on_template_error: true
cursor:
  timestamp:
    value: '[[.last_event.Event.timestamp]]'

If we modify the file like this, the ingest terminates. As soon as we include the Attribute split (without Object or Object.Attribute) the threatintel module will run forever.

I have saved the .ndjson from the terminating run, which might help to reproduce the issue.
The problematic MISP event has 27759 attributes and 3863 objects.

Best regards,
hti

[hti]

I managed to track down what is causing the issue.
There are several bugs involved. At least one is in the filebeat module.

The threatintel plugin assumes that the returned events from MISP are sorted by timestamp, as it will take the timestamp of the last event as its cursor criteria. However, the REST call made by the threatintel plugin does not include any sort instruction.
According to the MISP documentation (Automation and MISP API · User guide of MISP intelligence sharing platform), the REST call should include „order: Event.timestamp ASC“ , i.e. the config yml needs to be extended by

- set:
    target: body.order
    value: 'Event.timestamp ASC'

It's also required that the MISP developers fix their bug and honor the requested search order: Bug: REST API /events/restSearch ignores sort order · Issue #9359 · MISP/MISP · GitHub
And it is also required that the MISP developers fix their API documentation/implementation of timestamp filtering (Events restSearch API does not filter out events that has same timestamp value as filter · Issue #6803 · MISP/MISP · GitHub) or the threatintel plugin adapts the timestamp filter to "cursor.timestamp+1".

So at the moment MISP integration is completely broken.

Best regards,
hti