Can I use my own Threat Intel stored in plain txt file using filebeat module?


I have elastic SIEM 7.15 and have enabled threat intel filebeat module; it works very well however wondering if I can use my own threat intel gathered and stored in plain txt file? This comprises of IP Addresses, domains, URLs, hashes?

Can someone please guide me?

Blason R

Hi yes, definitly.

You will have to setup parsing for it though, either through logstash or through Elasticsearch ingest pipeline.

By configuring a file input you can read the txt file in with filebeat.

Agree what needs to be done to match the with the traffic? Is there any specific parsing done for inbuilt threat module.

I'd recommend reviewing the fields that are added with the threat intel module and adding the fields to your source data as needed. I've done this in my lab as follows -

Using the CINSscore threat intel feed, I ingest it through filebeat and use ingest pipelines to add the following fields:

event.module = threatintel

threatintel.indicator.provider = cinsscore
threatintel.indicator.type = ipv4-addr

I then rename the message field which only contains the IP to "threatintel.indicator.ip"

I'm also hashing the IP to generate a unique value for the _id field so I don't get duplicate intel records.

This works because the threatintel rules are running the following indicator matches -

(source.ip MATCHES threatintel.indicator.ip) OR (destination.ip MATCHES threatintel.indicator.ip)

and querying the following indicator index-

event.module:threatintel and (threatintel.indicator.ip:*)

I haven't looked into adding this into the filebeat module, but I am able to use filebeat to work off of the custom fields and indexes the module uses, so the module SIEM rules work.

Hey @n2x4

Would you mind sharing your Threat Intel filebeat config for reference purpose? Wanted to understand how do I match or write parsers to adhere with ECS schema.

Here's my config for my custom threat intel source (cinsscore). Just as a reference, this isn't using the ThreatIntel module - just a regular log source with the threat intel fields added and other stuff dropped that I don't need.

- type: log
  enabled: true

    - /home/elastic/cinsintel/*.txt
    threatintel.indicator.type: ipv4-addr
    service.type: threatintel
    threatintel.indicator.provider: cinsscore
    event.module: threatintel
  fields_under_root: true

  - rename:
        - from: "message"
          to: "threatintel.indicator.ip"
  - fingerprint:
      fields: ["threatintel.indicator.ip"]
      target_field: "@metadata._id"        
  - drop_fields:
      fields: ["agent.ephemeral_id", "", "agent.version", "ecs.version", "host.architecture", "host.containerized", "host.hostname", "", "host.ip", "host.mac", "", "host.os.codename", "", "host.os.kernel", "", "host.os.platform", "host.os.version", "input.type", "log.offset"] 

1 Like

Thanks mate!!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.