Filebeat MISP module refresh

Elastic Stack Beats
,
1

Is there a way to get the filebeat misp module to delete & re-create the index that stores the misp data in? I'm wanting to refresh this data hourly?

2

Can you elaborate more on this. What do you mean, what are you trying to achieve.

3

When you run the filebeat misp module it reads the threat intel data from misp and indexes it into an elasticsearch index and then exits.

I am trying to update the data in elasticsearch hourly with (updated) data from my misp instance rather than a once-off import...does that make sense?

4

Ah ok, thanks.

It looks like maybe there is a way to do this, it's just not documented - https://github.com/elastic/beats/issues/20524

5

Thanks Mark, I’ll try the var.interval setting. How does this work? Does it overwrite the index or just insert new docs?

6

I am not sure sorry, I only just came across that via an issue search!

7

A few thoughts:

  • see the MISP API docs where you can send in the body the time you want. EX.
    http_request_body: '{"publish_timestamp": "1d"}. If you set this with the interval, you might get what you want, but guessing it would still cause possible duplicates or missed events.
  • I have implemented this a few months ago and seems we didn't see it adding the same events. Possibly the ID is mapped to a unique ID so it replaces the existing record instead of creating a new one. I'd have to look closer at this.

Unfortunately, I can't back in to the test environment that I worked in to see what they finally settled on. I will update here if I get additional info.

8

Thanks Michelle - I've been playing with an ILM policy that deletes indexes after a period of time but this is wasteful and quirky.

Any idea on what arguments var.pagination takes or how to set this?

9

Why's that?

10

ILM policies are only checked every 10m and it appears that this can vary due to the multi-pass nature of ILM in my experience.

I might run it as a k8s cronJob so that I can do error handling if it appears that the misp module duplicates events / attributes

11

image
image1311×317 67.4 KB

I can confirm that the misp module duplicates events every run / interval @Michelle_Bennett

12

I've whipped up a kubernetes cronJob that deletes and re-creates the index every day https://github.com/hilt86/misp-importer

1 Like
13

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.