# Filebeat MISP module refresh **URL:** \ **Category:** Beats\ **Tags:** beats-module, filebeat\ **Created:** [November 22, 2020, 8:17pm UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281 "2020-11-22T20:17:49Z")\ **Posts on this page:** 13\ **Page:** 1
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [November 22, 2020, 8:17pm UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/1 "2020-11-22T20:17:49Z")
Is there a way to get the filebeat misp module to delete & re-create the index that stores the misp data in? I'm wanting to refresh this data hourly? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 22, 2020, 11:54pm UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/2 "2020-11-22T23:54:13Z")
> [@hilt86](#): > > I'm wanting to refresh this data hourly Can you elaborate more on this. What do you mean, what are you trying to achieve. ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [November 24, 2020, 1:04pm UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/3 "2020-11-24T13:04:42Z")
When you run the filebeat misp module it reads the threat intel data from misp and indexes it into an elasticsearch index and then exits. I am trying to update the data in elasticsearch hourly with (updated) data from my misp instance rather than a once-off import...does that make sense? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 25, 2020, 12:35am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/4 "2020-11-25T00:35:12Z")
Ah ok, thanks. It looks like **maybe** there is a way to do this, it's just not documented - [https://github.com/elastic/beats/issues/20524](https://github.com/elastic/beats/issues/20524) ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [November 25, 2020, 12:53am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/5 "2020-11-25T00:53:32Z")
Thanks Mark, I’ll try the var.interval setting. How does this work? Does it overwrite the index or just insert new docs? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [November 25, 2020, 12:55am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/6 "2020-11-25T00:55:01Z")
I am not sure sorry, I only just came across that via an issue search! ---
**Author:** ![Michelle\_Bennett](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/michelle_bennett/32/96142_2.png) [@Michelle\_Bennett](https://discuss.elastic.co/u/Michelle_Bennett)\ **Post date:** [December 3, 2020, 11:58pm UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/7 "2020-12-03T23:58:49Z")
A few thoughts: - see the MISP API docs where you can send in the body the time you want. EX. http\_request\_body: '{"publish\_timestamp": "1d"}. If you set this with the interval, you might get what you want, but guessing it would still cause possible duplicates or missed events. - I have implemented this a few months ago and seems we didn't see it adding the same events. Possibly the ID is mapped to a unique ID so it replaces the existing record instead of creating a new one. I'd have to look closer at this. Unfortunately, I can't back in to the test environment that I worked in to see what they finally settled on. I will update here if I get additional info. ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [December 4, 2020, 4:19am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/8 "2020-12-04T04:19:27Z")
Thanks Michelle - I've been playing with an ILM policy that deletes indexes after a period of time but this is wasteful and quirky. Any idea on what arguments var.pagination takes or how to set this? ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [December 4, 2020, 4:26am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/9 "2020-12-04T04:26:46Z")
> [@hilt86](#): > > I've been playing with an ILM policy that deletes indexes after a period of time but this is wasteful and quirky. Why's that? ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [December 4, 2020, 4:37am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/10 "2020-12-04T04:37:24Z")
ILM policies are only checked every 10m and it appears that this can vary due to the multi-pass nature of ILM in my experience. I might run it as a k8s cronJob so that I can do error handling if it appears that the misp module duplicates events / attributes ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [December 4, 2020, 5:14am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/11 "2020-12-04T05:14:00Z")
![image](https://us1.discourse-cdn.com/elastic/original/3X/d/9/d9e235c6ed94b3b44fc1d2e2bbfb77ccc211c2ab.png) I can confirm that the misp module duplicates events every run / interval @Michelle_Bennett ---
**Author:** ![hilt86](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilt86/32/20308_2.png) [@hilt86](https://discuss.elastic.co/u/hilt86)\ **Post date:** [December 9, 2020, 1:33am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/12 "2020-12-09T01:33:26Z")
I've whipped up a kubernetes cronJob that deletes and re-creates the index every day [https://github.com/hilt86/misp-importer](https://github.com/hilt86/misp-importer) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 6, 2021, 3:33am UTC](https://discuss.elastic.co/t/filebeat-misp-module-refresh/256281/13 "2021-01-06T03:33:37Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.