# Doubts about Filebeat Threat Intel Module \[7.12.0\] **URL:** \ **Category:** Beats\ **Tags:** beats-module, filebeat\ **Created:** [April 21, 2021, 9:15am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810 "2021-04-21T09:15:43Z")\ **Posts on this page:** 15\ **Page:** 1
**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Post date:** [April 21, 2021, 9:15am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/1 "2021-04-21T09:15:43Z")
Hi, everyone I have been testing with Filebeat [Threat Intel module](https://www.elastic.co/guide/en/beats/filebeat/7.x/filebeat-module-threatintel.html) in order to get events from **MISP**. I have some doubts regarding configuration: 1. I have to disable every input plugin because **they are enabled by default**. Should they be disabled by default ? 2. From my point of view it would be a good feature add GeoIP for **threatintel.indicator.ip** in order to know the location. 3. It seems that when there are not events in the time range the plugins shows an **Error**. Should be a **Warning**? 4. How the module deals with duplicates? Here you are my config: ```auto filebeat.modules: - module: threatintel abuseurl.enabled: false abusemalware.enabled: false malwarebazaar.enabled: false anomali.enabled: false otx.enabled: false misp: enabled: true var.input: httpjson var.url: "https://:443/events/restSearch" var.api_token: "" var.first_interval: 24h var.interval: 12h var.ssl.verification_mode: none processors: - convert: fields: - from: "@metadata.pipeline" to: "fields.pipeline" logging.level: warning output.kafka: hosts: [''] topic: misp client_id: misp ``` Thanks in advance, Rodrigo ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 22, 2021, 1:11am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/2 "2021-04-22T01:11:32Z")
Currently when u enable a module, all the filesets are enabled by default as well. For the GeoIP, that could be added easily. I'd have to see about the error vs warning since I haven't touched the threat intel module yet. ---
**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Post date:** [April 22, 2021, 5:52am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/3 "2021-04-22T05:52:08Z")
Hi, @legoguy1000 I have found duplicated events after the second interval. I would like to know how this module deals with it. Thanks in advance ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [April 22, 2021, 12:04pm UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/4 "2021-04-22T12:04:29Z")
> [@RdrgPorto](#): > > I have to disable every input plugin because **they are enabled by default**. Should they be disabled by default ? Unfortunately they are not disabled by default. See this issue: [[Filebeat][proposal] Disable filesets that are not configured · Issue #17256 · elastic/beats · GitHub](https://github.com/elastic/beats/issues/17256) ---
**Author:** ![legoguy1000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/legoguy1000/32/54301_2.png) [@legoguy1000](https://discuss.elastic.co/u/legoguy1000)\ **Post date:** [April 22, 2021, 12:20pm UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/5 "2021-04-22T12:20:28Z")
Regarding the duplicate events, I have seen a discussion about this before. @andrewkroh check me on this but looking at the `threatintel.misp` module vs the deprecated `misp.threat` module, the `misp.threat`module has as part of its local processing `evt.Put("@metadata._id", evt.Get("event.id"));`. The `threatintel.misp` module doesn't seem to set the `@metadata._id` at all which would allow duplicates. Looks like the `threatintel.abuseurl` module has ```auto - fingerprint: fields: ["json.id"] target_field: "@metadata._id" ``` ---
**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\ **Post date:** [April 22, 2021, 12:58pm UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/6 "2021-04-22T12:58:10Z")
It looks like old version uses the `Event.Attributes.uuid` as the document `_id`. The description of uuid is at [misp-rfc/raw.md.txt at master · MISP/misp-rfc · GitHub](https://github.com/MISP/misp-rfc/blob/master/misp-core-format/raw.md.txt#L175). I think setting \_id is a good safety mechanism. I am concerned that the same `uuid` is used for updates. This means that any update to a MISP record will cause an \_id collision. And because Filebeat users are typically assigned the `create_doc` privilege this will cause the updates to fail. - [Grant privileges and roles needed for publishing | Filebeat Reference [7.12] | Elastic](https://www.elastic.co/guide/en/beats/filebeat/7.12/privileges-to-publish-events.html) - [Security privileges | Elasticsearch Guide [7.12] | Elastic](https://www.elastic.co/guide/en/elasticsearch/reference/current/security-privileges.html#privileges-list-indices) I don't know if the intended behavior of setting \_id was to update the MISP record in Elasticsearch or just to prevent duplicates. If the intent was to prevent duplicates then the new module could go the `fingerprint` route and incorporate the uuid and misp [`timestamp`](https://github.com/MISP/misp-rfc/blob/master/misp-core-format/raw.md.txt#L742) since it should change if the record is updated. ---
**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Post date:** [April 22, 2021, 1:12pm UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/7 "2021-04-22T13:12:17Z")
Hi, @andrewkroh I think it could work. Thanks 🙂, Regards ---
**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Post date:** [April 23, 2021, 5:35am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/8 "2021-04-23T05:35:27Z")
Hi, @andrewkroh Should we create an issue on GitHub ? Regards ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [April 24, 2021, 9:26am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/9 "2021-04-24T09:26:06Z")
Hello @RdrgPorto, @andrewkroh @legoguy1000, Hopefully I can shed some light on some of these questions, please let me know if there is something I didn't cover. 1. For the question around enabling/disabling, it seems to have been covered already by Andrew, and its something that is of course up for discussion, following that issue seems to be the best approach. 2. The GeoIP processor is absolutely a valid point for the threatintel.indicator field, I will bring this up and see if we want to enable this as well. 3. Changing the logged message from error to debug has been implemented in 7.12.1+7.13 I believe, I will check if MISP is using that new option as well, it should now only be logged if debug is enabled. The reason is that there will always be an empty response for modules that uses pagination, once it reaches the last page, so leaving it at warning level is still too high, as we only want to show this during debugging. 4. The duplication between the old and new misp module is a bit different, and the new MISP fileset which is part of the threatintel module should be overwriting the updates because of the op\_type that is set, it also uses a different API than the old MISP module, as the old uses attribute/restSearch, while the new uses event/restSearch, the format of which the events come back is a bit different. When we receive any of the raw events, they all look like this: [beats/misp\_sample.ndjson.log at master · elastic/beats · GitHub](https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/threatintel/misp/test/misp_sample.ndjson.log) All attributes related to an Event will be split up into its own document, each Event has its own uuid, while each attribute also has its own uuid. The processor that is used by the module, documented here: [beats/config.yml at master · elastic/beats · GitHub](https://github.com/elastic/beats/blob/master/x-pack/filebeat/module/threatintel/misp/config/config.yml#L62) Will first set the document\_id to the uuid of the attribute: ``` - decode_json_fields: fields: [message] document_id: Event.Attribute.uuid target: json ``` After that, to allow documents to be overwritten, so that instead of duplicates, the old document is replaced by a new one, we overwrite the default op\_type field of the metadata: ``` - script: lang: javascript id: my_filter source: > function process(event) { event.Put("@metadata.op_type", "index"); } ``` The intended behavior here is that duplicated events should be overwriting the existing one, instead of creating a second event. Once filebeat has received the response from the API, it will take the timestamp from the newest event it receieved, and use that as a filter for the API on any new API calls after that, so that we do not end up receiving the same events multiple times, causing constant overwrites for example. This is done by setting cursor.timestamp to the request body, and if this is the first time filebeat starts up and cursor.timestamp do not exist it will look back the configured value on `var.first_interval` in your configuration file. ``` - set: target: body.timestamp value: '[[.cursor.timestamp]]' default: '[[formatDate (now (parseDuration "-{{ .first_interval }}")) "UnixDate"]]' ``` Cursor.timestamp is magically set based on the timestamp from the last event like so: ``` cursor: timestamp: value: '[[.last_event.Event.timestamp]]' ``` Hopefully this clear things up 🙂 And thanks for all the feedback, keep it coming because we are always trying to make the modules better! ---
**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Post date:** [April 26, 2021, 7:16am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/10 "2021-04-26T07:16:27Z")
> [@Marius\_Iversen](#): > > ```auto > document_id: Event.Attribute.uuid > > ``` Hi, @Marius_Iversen I have been testing with this module and I have found out **duplicated events**. As you can see in the next screenshot, the same attribute is duplicated in **Elasticsearch** : ![01-es](https://us1.discourse-cdn.com/elastic/original/3X/7/9/79fd6a1dcd8209a409c4cba0c08d4907404b61cb.png) If I correctly understand your explanation, the **Elasticsearch document id** should be the **MISP attribute uuid**. Nevertheless, I see that **\_id** does not match **MIPS uuid**. ![02-es](https://us1.discourse-cdn.com/elastic/original/3X/2/a/2abf005d881773d0c5eaa3458e2a3ad438f4163a.png) Moreover, if I execute **Filebeat module** (`Filebeat 7.12.0 with Elasticsearch 7.9.2`) several times, it duplicates the results every time not taking into account the **timestamp** as you explained. Here you are some screenshots of **MISP** : ![01-misp](https://us1.discourse-cdn.com/elastic/original/3X/1/e/1ed678af811d87b3bd704a9e09190d5ed65ad8d5.png) ![02.misp](https://us1.discourse-cdn.com/elastic/original/3X/2/9/292607058b56cce3ac64d976746e82226e9b0c29.png) Thanks in advance, Regards ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [April 26, 2021, 1:29pm UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/11 "2021-04-26T13:29:23Z")
Using different versions might have some undesired effects, however I do believe that's not what is causing your issues right now. I will see if I can reproduce it and come up with a fix if needed, will let you know how it goes. ---
**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Post date:** [April 26, 2021, 3:16pm UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/12 "2021-04-26T15:16:33Z")
Hi, @Marius_Iversen Thank you very much 😃 Regards ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [May 10, 2021, 12:47pm UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/15 "2021-05-10T12:47:49Z")
Just removed my older posts, as they would have been conflicting with the results. We ran some tests with and without the uuid setting for the filebeat module against our own misp setup and saw that the expected behavior did indeed happen, so this seems to be something specific to your setup: Below is the picture, the first event is with the UUID option removed, and the rest is with the UUID like it is today, the field is the `_id` of the documents. ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cb348b544e3c3527e5bde9488be15d98f74b41ba.png) Is this from a beat that has been updated multiple times? Maybe it has some conflicting yml files from older versions? The ID fields from your screenshots is not what you want to match: `threatintel.misp.uuid` is the `Event UUID`. `threatinte.misp.id` is the `Event ID`. `threatintel.misp.attribute.id` is the `Attribute ID`. And the `document._id`should be the UUID of the Attribute, though in your case its not for some reason. Can you try to set a file output in filebeat and see how the output files looks like? While most of the parsing is gone, since that happens on ES, you should have a proper \_id field in the @metadata section of the resulting output. ---
**Author:** ![RdrgPorto](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rdrgporto/32/13278_2.png) [@RdrgPorto](https://discuss.elastic.co/u/RdrgPorto)\ **Post date:** [May 27, 2021, 11:33am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/16 "2021-05-27T11:33:04Z")
Hi, @Marius_Iversen I have tested with **Elastic Stack 7.12.1** ( **Elasticsearch** , **Kibana** & **Filebeat** ) and it works. I think that this issue is related to **Elastic Stack 7.9.2**. Thanks for all, Regards ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 8:29am UTC](https://discuss.elastic.co/t/doubts-about-filebeat-threat-intel-module-7-12-0/270810/17 "2022-11-04T08:29:16Z")