Add module threatintel to filebeat

Elastic Stack Beats
,
1

Hi, I'm adding threatintel module to filebeat , but when I restart filebeat service it failed, I get this error : Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

2

Hello there!
Please paste your filebeat.yml and modules.d/threatintel.yml (using </> tag of course).

3 
- module: threatintel
  abusemalware:
    enabled: true
    var.input: httpjson
    var.url: https://urlhaus-api.abuse.ch/v1/payloads/recent/
    var.interval: 60m
4

I had already enabled threat intel module : sudo filebeat modules enable threatintel

5

Three questions:

  1. Is filebeat starting properly when you disable this module?
  2. Is this complete module config file?
  3. Could you please paste filebeat.yml (covering your sensitive data)?
6

When i disable this module filebeat is starting normaly , for the moment this all config file

7

in filebeat.yml it seems that it send logs to kafka , do you think may cause issue ??

8

Share your entire filebeat.yml.

If your filebeat already has a kafka output, you cannot configure other outputs, filebeat only supports one output.

9 
filebeat:
  config:
    inputs:
      enabled: true
      path: /etc/filebeat/inputs.d/*.yml
    modules:
      enabled: true
      path: /etc/filebeat/modules.d/*.yml
processors:
- add_fields:
    fields:
      environment: test
      groups:
      - auditbeat_hosts
    target: inventory
- add_fields:
    fields:
      criticity: low
    target: service

##############################################################
############################# Output ##########################################

output:
  kafka:
    codec.json:
      pretty: false
    compression: gzip
    hosts:
    - X.X.X.X:9092
    max_message_bytes: 1000000
    partition.round_robin:
      reachable_only: true
    required_acks: 1
    topic: filebeat
10

And what is the error you get? You need to provide the log error that you receive when you start filebeat with the threatintel module activate, the message you shared in your first post is not enough to understand what could be happening.

11 
 filebeat.service: Service RestartSec=100ms expired, scheduling restart.
Aug 19 12:37:33 test systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 5.
Aug 19 12:37:33 test systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Aug 19 12:37:33 test systemd[1]: filebeat.service: Start request repeated too quickly.
Aug 19 12:37:33 test  systemd[1]: filebeat.service: Failed with result 'exit-code'.
Aug 19 12:37:33 test  systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
12

This also does not help, you need to check the systems logs, /var/log/messages or /var/log/syslog depending on the linux distribution.

There you will find the reason why the filebeat service could not start.

13 
2022-08-19T15:20:15.569+0200	ERROR	instance/beat.go:958	Exiting: Index management requested but the Elasticsearch output is not configured/enabled
Exiting: Index management requested but the Elasticsearch output is not configured/enabled
14

Usually this error shows when in filebeat.yml (or other conf file) you enabled this lines:

#setup.template.settings:
  #index.number_of_shards: 1
  #index.codec: best_compression
  #_source.enabled: false

but Elasticsearch output is disabled.

15

finaly , I find these lines in filebeat.reference.yml but there are disabled

16

Actually , I have this error

Exiting: Failed to start crawler: creating module reloader failed: fileset threatintel/abusemalware is configured but doesn't exist
17

This file is only a reference, not a real config file used to set things up and run filebeat.

You should add to you module conf file:

  abusemalware:
    enabled: false

if you don't want to use it.

18

the main purpose is to add threat intel module , this module use abuseurl (filesets) so i think it must be enabled

19

I have this error ,when I restart filebeat

[34492]: 2022-08-23T10:09:01.575+0200        INFO        log/input.go:152        Configured paths: [/var/log/auth.log* /var/log/secure*]
Aug 23 10:09:01[34492]: 2022-08-23T10:09:01.575+0200        INFO        log/input.go:152        Configured paths: [/var/log/messages* /var/log/syslog*]
Aug 23 10:09:01[34492]: 2022-08-23T10:09:01.575+0200        INFO        beater/crawler.go:148        Stopping Crawler
Aug 23 10:09:01 [34492]: 2022-08-23T10:09:01.575+0200        INFO        beater/crawler.go:158        Stopping 0 inputs
Aug 23 10:09:01 [34492]: 2022-08-23T10:09:01.575+0200        INFO        beater/crawler.go:178        Crawler stopped
Aug 23 10:09:01[34492]: 2022-08-23T10:09:01.575+0200        INFO        registrar/registrar.go:367        Stopping Registrar
Aug 23 10:09:01 [34492]: 2022-08-23T10:09:01.575+0200        INFO        registrar/registrar.go:293        Ending Registrar
Aug 23 10:09:01[34492]: 2022-08-23T10:09:01.580+0200        INFO        [monitoring]        log/log.go:153        Total non-zero metrics        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":24}},"total":{"ticks":40,"t
Aug 23 10:09:01[34492]: 2022-08-23T10:09:01.580+0200        INFO        [monitoring]        log/log.go:154        Uptime: 35.592538ms
Aug 23 10:09:01 [34492]: 2022-08-23T10:09:01.580+0200        INFO        [monitoring]        log/log.go:131        Stopping metrics logging.
Aug 23 10:09:01[34492]: 2022-08-23T10:09:01.580+0200        INFO        instance/beat.go:469        filebeat stopped.
Aug 23 10:09:01 [34492]: 2022-08-23T10:09:01.580+0200        ERROR        instance/beat.go:958        Exiting: Failed to start crawler: creating module reloader failed: fileset threatintel/abuseurl is configured but doesn't exist
Aug 23 10:09:01 [34492]: Exiting: Failed to start crawler: creating module reloader failed: fileset threatintel/abuseurl is configured but doesn't exist
20

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.