Filebeat MISP module refresh

Is there a way to get the filebeat misp module to delete & re-create the index that stores the misp data in? I'm wanting to refresh this data hourly?

Can you elaborate more on this. What do you mean, what are you trying to achieve.

When you run the filebeat misp module it reads the threat intel data from misp and indexes it into an elasticsearch index and then exits.

I am trying to update the data in elasticsearch hourly with (updated) data from my misp instance rather than a once-off import...does that make sense?

Ah ok, thanks.

It looks like maybe there is a way to do this, it's just not documented - https://github.com/elastic/beats/issues/20524

Thanks Mark, I’ll try the var.interval setting. How does this work? Does it overwrite the index or just insert new docs?

I am not sure sorry, I only just came across that via an issue search!

A few thoughts:

  • see the MISP API docs where you can send in the body the time you want. EX.
    http_request_body: '{"publish_timestamp": "1d"}. If you set this with the interval, you might get what you want, but guessing it would still cause possible duplicates or missed events.
  • I have implemented this a few months ago and seems we didn't see it adding the same events. Possibly the ID is mapped to a unique ID so it replaces the existing record instead of creating a new one. I'd have to look closer at this.

Unfortunately, I can't back in to the test environment that I worked in to see what they finally settled on. I will update here if I get additional info.

Thanks Michelle - I've been playing with an ILM policy that deletes indexes after a period of time but this is wasteful and quirky.

Any idea on what arguments var.pagination takes or how to set this?

Why's that?

ILM policies are only checked every 10m and it appears that this can vary due to the multi-pass nature of ILM in my experience.

I might run it as a k8s cronJob so that I can do error handling if it appears that the misp module duplicates events / attributes

I can confirm that the misp module duplicates events every run / interval @Michelle_Bennett