Threat Intel Without Elastic Agent

Elastic Security
1

Hi!

I'm trying to use Elastic's Threat Intelligence modules without having to use any kind of agents.

My elastic server is receiving parsed logs from logstash into specific indices. These indices have the fields 'ip' and 'from'; 'ip' contains IP address and 'from' contains the domain name of an email id. I want to write an elastic rule to determine if these two fields come from malicious sources, for which i want to use elastic's threat intelligence module (or MISP integration). Ideally without any elastic agent.

Any inputs on how to go about? Thanks in advance!

2

Hello @aadityar

You can use the Threat Intel module from Filebeat.

I am also sharing our doc page with more information about Threat Intel integrations:

Thanks!

3

Thanks @lcamargo !

I've looked at the filebeat method and i've tried to implement it, but there seems to be a configuration issue. No data is appearing in the dashboard.

Inspite of enabling the MISP fileset in the threatintel module, I receive this error message:

Exiting: module threatintel is configured but has no enabled filesets

The API key and URL seem to be working fine, since I've checked them using curl commands. Could you help on this?

4

Hey @aadityar

How did you configure the module? Have you enabled it?
Please share the module file configuration and the output for filebeat modules list.

5 
filebeat modules list

Enabled:
misp
threatintel
Disabled:
activemq
apache
auditd
aws
awsfargate
azure
cef
checkpoint
cisco
coredns
crowdstrike
cyberarkpas
elasticsearch
envoyproxy
fortinet
gcp
google_workspace
haproxy
ibmmq
icinga
iis
iptables
juniper
kafka
kibana
logstash
microsoft
mongodb
mssql
mysql
mysqlenterprise
nats
netflow
nginx
o365
okta
oracle
osquery
panw
pensando
postgresql
rabbitmq
redis
salesforce
santa
snyk
sophos
suricata
system
traefik
zeek
zookeeper
zoom

Running this command generates this error

filebeat setup --pipelines --modules threatintel
Exiting: module threatintel is configured but has no enabled filesets

Inspite of enabling MISP in filebeat/modules.d/threatintel.yml

misp:
    enabled: true

    # Input used for ingesting threat intel data, defaults to JSON.
    var.input: httpjson

    # The URL of the MISP instance, should end with "/events/restSearch".
    var.url: https://<server-ip>/events/restSearch

    # The authentication token used to contact the MISP API. Found when looking at user account in the MISP UI.
    var.api_token: <api-token>

    # Configures the type of SSL verification done, if MISP is running on self signed certificates
    # then the certificate would either need to be trusted, or verification_mode set to none.
    var.ssl.verification_mode: none

    # Optional filters that can be applied to the API for filtering out results. This should support the majority of fields in a MISP context.
    # For examples please reference the filebeat module documentation.
    #var.filters:
    #  - threat_level: [4, 5]
    #  - to_ids: true

    # How far back to look once the beat starts up for the first time, the value has to be in hours. Each request afterwards will filter on any event newer
    # than the last event that was already ingested.
    var.first_interval: 300h

    # The interval to poll the API for updates.
    var.interval: 5m

hiding url and key here, they are populated in actual file

6

I reinstalled filebeat, these are a portion of the logs when i run journalctl -u filebeat

Jan 08 08:10:17 elastic filebeat[234212]: {"log.level":"info","@timestamp":"2025-01-08T08:10:17.881Z","log.logger":"modules","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/fileset.newModuleRegistry","file.name":"fileset/modules.go","file.line":136},"message":"Enabled modules/filesets: threatintel (misp)","service.name":"filebeat","ecs.version":"1.6.0"}
Jan 08 08:10:17 elastic filebeat[234212]: {"log.level":"warn","@timestamp":"2025-01-08T08:10:17.882Z","log.logger":"input","log.origin":{"function":"github.com/elastic/beats/v7/filebeat/input/v2/compat.(*factory).CheckConfig","file.name":"compat/compat.go","file.line":81},"message":"input V2 factory.CheckConfig failed to clone config before checking it. Original config will be checked, it might trigger an input duplication warning: failed to get 'id': missing field accessing 'id'","service.name":"filebeat","original_config":"{\n  \"_fileset_name\": \"misp\",\n  \"_module_name\": \"threatintel\",\n  \"cursor\": {\n    \"timestamp\": {\n      \"value\": \"[[.last_event.Event.timestamp]]\"\n    }\n  },\n  \"interval\": \"5m\",\n  \"path\": {\n    \"config\": \"/usr/share/filebeat\",\n    \"data\": \"/usr/share/filebeat/data\",\n    \"home\": \"/usr/share/filebeat\",\n    \"logs\": \"/usr/share/filebeat/logs\"\n  },\n  \"pipeline\": \"filebeat-8.17.0-threatintel-misp-pipeline\",\n  \"processors\": [\n    {\n      \"add_locale\": null\n    },\n    {\n      \"add_fields\": {\n        \"fields\": {\n          \"ecs\": {\n            \"version\": \"1.12.0\"\n          }\n        },\n        \"target\": \"\"\n      }\n    }\n  ],\n  \"publisher_pipeline\": {\n    \"disable_host\": true\n  },\n  \"request\": {\n    \"body\": null,\n    \"method\": \"POST\",\n    \"transforms\": [\n      {\n        \"set\": {\n          \"target\": \"header.Authorization\",\n

The dashboards arent reflecting anything still