Can I use my own Threat Intel stored in plain txt file using filebeat module?

[Blason (R)]

Hi,

I have elastic SIEM 7.15 and have enabled threat intel filebeat module; it works very well however wondering if I can use my own threat intel gathered and stored in plain txt file? This comprises of IP Addresses, domains, URLs, hashes?

Can someone please guide me?

TIA
Blason R

[sholzhauer]

Hi yes, definitly.

You will have to setup parsing for it though, either through logstash or through Elasticsearch ingest pipeline.

By configuring a file input you can read the txt file in with filebeat.

[Blason (R)]

Agree what needs to be done to match the with the traffic? Is there any specific parsing done for inbuilt threat module.

[n2x4 (Nate)]

I'd recommend reviewing the fields that are added with the threat intel module and adding the fields to your source data as needed. I've done this in my lab as follows -

Using the CINSscore threat intel feed, I ingest it through filebeat and use ingest pipelines to add the following fields:

event.module = threatintel

threatintel.indicator.provider = cinsscore
	
threatintel.indicator.type = ipv4-addr

I then rename the message field which only contains the IP to "threatintel.indicator.ip"

I'm also hashing the IP to generate a unique value for the _id field so I don't get duplicate intel records.

This works because the threatintel rules are running the following indicator matches -

(source.ip MATCHES threatintel.indicator.ip) OR (destination.ip MATCHES threatintel.indicator.ip)

and querying the following indicator index-

event.module:threatintel and (threatintel.indicator.ip:*)

I haven't looked into adding this into the filebeat module, but I am able to use filebeat to work off of the custom fields and indexes the module uses, so the module SIEM rules work.

[Blason (R)]

Hey @n2x4

Would you mind sharing your Threat Intel filebeat config for reference purpose? Wanted to understand how do I match or write parsers to adhere with ECS schema.

[n2x4 (Nate)]

Here's my config for my custom threat intel source (cinsscore). Just as a reference, this isn't using the ThreatIntel module - just a regular log source with the threat intel fields added and other stuff dropped that I don't need.

- type: log
  enabled: true

  paths:
    - /home/elastic/cinsintel/*.txt
  fields:
    threatintel.indicator.type: ipv4-addr
    service.type: threatintel
    threatintel.indicator.provider: cinsscore
    event.module: threatintel
  fields_under_root: true

  processors:
  - rename:
      fields:
        - from: "message"
          to: "threatintel.indicator.ip"
  - fingerprint:
      fields: ["threatintel.indicator.ip"]
      target_field: "@metadata._id"        
  - drop_fields:
      fields: ["agent.ephemeral_id", "agent.id", "agent.version", "ecs.version", "host.architecture", "host.containerized", "host.hostname", "host.id", "host.ip", "host.mac", "host.name", "host.os.codename", "host.os.family", "host.os.kernel", "host.os.name", "host.os.platform", "host.os.version", "input.type", "log.offset"] 

[Blason (R)]

Thanks mate!!