# \[ URLHaus threat intelligence \]: create a new rule

**URL:** <https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857>\
**Category:** SIEM\
**Tags:** elastic-stack-alerting\
**Created:** [January 12, 2021, 3:49pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857 "2021-01-12T15:49:44Z")\
**Posts on this page:** 19\
**Page:** 1

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 12, 2021, 3:49pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/1 "2021-01-12T15:49:44Z")

</div>

Hello,

I am using logsatsh to enrich my SIEM with malicious Urls that I downloaded from URLHaus,  
and I wanna create a rule that send an alert everytime a user connect to one of the malicious urls.  
The problem is that I didn't find a field in packetbeat which contains the full url, so I couldn't use the `Match` alert

for example:  
Malicious url:

```auto
http://www.davidephoto.it/GsnIO/

```

and packetbeat fields:

```auto
**dns**
dns.question.name: www.davidephoto.it
dns.question.etld_plus_one: davidephoto.it
dns.question.registered_domain: davidephoto.it
dns.question.subdomain: www
dns.question.top_level_domain: it

**tls**
destination.domain: www.davidephoto.it
server.domain: www.davidephoto.it
tls.client.server_name: www.davidephoto.it

```

Could you tell me if I can use something like `contain` as non of the fields `Match` completly to the url ? or any other solution

Thanks for your help

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 13, 2021, 1:51am UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/2 "2021-01-13T01:51:45Z")

</div>

Is this specifically in the SIEM app?

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 13, 2021, 8:02am UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/3 "2021-01-13T08:02:06Z")

</div>

it's for security reaspns, to detect malicious connections, so yes it's for SIEM utilisation

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 13, 2021, 9:35pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/4 "2021-01-13T21:35:04Z")

</div>

Ok, if it's in the SIEM app though I will move the topic to that category.

---

<div class="post-metadata">

**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\
**Post date:** [January 14, 2021, 1:03pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/5 "2021-01-14T13:03:46Z")

</div>

Hi @TheHunter1,

Elastic Common Schema[1] defines a field for storing the complete URL as `url.full`, as documented [here](https://www.elastic.co/guide/en/ecs/current/ecs-url.html#field-url-full).

Packetbeat populates the `url.full` field for http events (i.e., where `network.protocol: "http"`).

`url.full` is not populated in DNS or TLS events from Packetbeat, because the URL is either not present or not visible in those protocols.

Perhaps instead of using a "URL" threat list, you might consider a "DNS" or "Domain"-based threat list, which you will be able to match against your DNS events. You could then compare the `dns.question.name` field in your events against the values in this list.

Here's an interesting example of such as list ([https://feeds.alphasoc.net/ryuk.txt](https://feeds.alphasoc.net/ryuk.txt)) that you might want to investigate.

[1]  
I want to ensure that you are aware of [Elastic Common Schema](https://www.elastic.co/what-is/ecs) (ECS).

The Elastic SIEM/Security app, including its detection rules, signals, and detection alerts, _requires_ your data to be indexed in an ECS-compliant format. [ECS](https://www.elastic.co/guide/en/ecs/current/ecs-reference.html) is an open source, community-developed schema that specifies field names and Elasticsearch data types for each field, and provides descriptions and example usage.

The easiest way to get your data in ECS-compliant format is to use an Elastic-supplied beat module, (e.g., [filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-modules-overview.html) or Elastic Agent integration), which will ingest and index your data in an ECS-compliant format. Elastic provides a growing list of these integrations that you can find on our [Integrations page](https://www.elastic.co/integrations?solution=security).

If you're using a custom data ingestion method (like your use of Logstash), or one provided by a third-party, then you may need to convert your data so that it is in an ECS-compliant format before you can use the SIEM/security app. This can be done by creating your own beat/module, or your own Logstash configuration for each data source, which will convert your data to ECS during the ingestion process.

[General guidelines](https://www.elastic.co/guide/en/ecs/current/ecs-guidelines.html) for creating ECS-compliant data:

1. Each indexed document (e.g., your log, event, etc.) MUST have the `@timestamp` field.
2. Your index mapping template must specify the [Elasticsearch field data type](https://www.elastic.co/guide/en/elasticsearch/reference/7.9/mapping-types.html) for each field as defined by ECS. For example, your `@timestamp` field must be of the `date` field data type, etc.. This ensures that there will not be any mapping conflicts in your indices.
3. The original fields from your log/event SHOULD be copied/renamed/converted to the corresponding ECS-defined field name and data type.
4. Additional ECS fields, such as the [ECS Categorization fields](https://www.elastic.co/guide/en/ecs/current/ecs-category-field-values-reference.html) SHOULD be populated for each log/event, to allow proper inclusion of your data into dashboards and detection rules.

A list of the specific ECS fields used by the SIEM/Security app is provided in this [reference](https://www.elastic.co/guide/en/security/current/siem-field-reference.html).

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 14, 2021, 4:57pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/6 "2021-01-14T16:57:29Z")

</div>

Thanks for all your exaplanation @Mike_Paquette, it was so clair  
so as I can understand I can't use the rule `Match` in this case as there is no field that contains the full URL in DNS and TLS  
is there a posibility to create a rule with something like `IN` to match just some of the full URL  
Exemple:

The `full URL` is:

```auto
http://www.davidephoto.it/GsnIO/

```

and we can see in **DNS** `dns.question.name` and **TLS** `destination.domain:` they both contain the url `www.davidephoto.it`

The idea is to say if: `dns.question.name` OR `destination.domain:` **Contain** `full.url` then `alerte`

we can see that `www.davidephoto.it` **IS IN** `http://www.davidephoto.it/GsnIO/`

Hope you can understand what I am meaninig 😅

---

<div class="post-metadata">

**Author:** ![hilo21](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilo21/32/66272_2.png) [@hilo21](https://discuss.elastic.co/u/hilo21)\
**Post date:** [January 14, 2021, 8:56pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/7 "2021-01-14T20:56:55Z")

</div>

Hello,

Can you tell me please what kind of alerting mechanism you're using ?

- Watcher
- SIEM Detections

Also is the URLhaus events stored in a different index than your packetbeat or are these fields on the same Document ?

Thank you

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 15, 2021, 8:03am UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/8 "2021-01-15T08:03:46Z")

</div>

Thanks for your answer @hilo21,

So I want if possibile to create a rule from the SIEM detections without using a script to create a watcher, unless if there is no way to do it with the SIEM rules.

and for your second question, I configured logsatsh to download the URLHaus database each 24h and store it in a dedicated index just for URLHaus.

---

<div class="post-metadata">

**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\
**Post date:** [January 15, 2021, 2:12pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/9 "2021-01-15T14:12:54Z")

</div>

Hi @TheHunter1 To further clarify your question, I think you are asking if a SIEM/Security App detection rule can MATCH a _keyword_ field such as `dns.question.name` or `destination.domain` when the keyword field value is _contained in_ the URL field. Like this (shown with the Security App Indicator Match rule type):

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/5/f54cd2921959b1373ec9d5520c9a0295d271b690.jpeg)

Is that your question?

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 15, 2021, 3:51pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/10 "2021-01-15T15:51:06Z")

</div>

Thanks for your answer @Mike_Paquette,  
But by doing that I think that it won't work as there is not a 100% `MATCH` between the 2 urls.

The threat url: `http://www.davidephoto.it/GsnIO/`  
dns.question.name in packetbeat field: `www.davidephoto.it`

I think the`Match` will work only if there is a 100% match between the 2 fields, isn't it ?

---

<div class="post-metadata">

**Author:** ![hilo21](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hilo21/32/66272_2.png) [@hilo21](https://discuss.elastic.co/u/hilo21)\
**Post date:** [January 15, 2021, 5:05pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/11 "2021-01-15T17:05:10Z")

</div>

One way of manh to do this is to use elasticsearch filter plugin in packetbeat logstash pipeline to send a DSL query to elasticsearch for whatever index you want. I guess whith some testing you would be able to do it. Try wildcard query maybe it will work. Never used it in this use case but you can try this workaround.  
But what you should be doing is parsing the urls to get site, etld and tld out of them this way your data would be a little bit cleaner and flexible for many use cases.

---

<div class="post-metadata">

**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\
**Post date:** [January 18, 2021, 12:14pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/12 "2021-01-18T12:14:50Z")

</div>

Thanks @TheHunter1 I was just trying to understand your question 🙂

You are correct that the MATCH implemented in the Indicator Match rule type is an exact match, and will NOT address your specific request.

For now, you'll need to have separate indicator match rules for each type of threat indicator you want to compare against.

Please let us know how you decide to proceed. Thanks!

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 18, 2021, 1:20pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/13 "2021-01-18T13:20:55Z")

</div>

Hello again,

So I did as you and @hilo21 suggested, I used a`grok` filter, and separated the indicators.  
The grok filter I used is:

```auto
    grok {
      match => { "url" => "%{URIPROTO:uri_proto}:\/\/(%{IP:destination.ip}:%{NUMBER:destination.port})?(%{URIHOST:domain})?\/%{GREEDYDATA:uri_para}" }
    }

```

( somtimes the indicator is an IP address and port and sometimes it's URL, that why my filter looks like that)

and my rule is like that:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/2/e2727321b005b1f7ab6f93f2720ab58a3c679b54.png)

the only problem is that I am getting this error while creating the rule so I couldn't test if it's working correctly :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/2/82e2afc81dc6d507eb4389557861779ba228b1ba.png)

So I created a new Topic here :

> [@\[Creating new rule \]: ERROR Authentication using apikey failed - api key has been invalidated](https://discuss.elastic.co/t/creating-new-rule-error-authentication-using-apikey-failed-api-key-has-been-invalidated/261407):
>
> Hello, I am using elasticsearch, kibana and beats version 7.10.1 with a Trial license, I am trying to create a new Match rule and in my elasticsearch node I am getting this error: [2021-01-18T11:33:18,032][WARN][o.e.x.s.a.AuthenticationService] [DATA-03] Authentication using apikey failed - api key has been invalidated [2021-01-18T11:33:18,035][WARN][o.e.x.s.a.AuthenticationService] [DATA-03] Authentication using apikey failed - api key has been invalidated and in kibana interface: Rule …

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 19, 2021, 9:31am UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/14 "2021-01-19T09:31:07Z")

</div>

Hello again @Mike_Paquette, @hilo21 ,  
So the problem of the creation of the rule has been solved,

for the `http` URL It's working fine as I am using the `url.full` of packetbeat, but for `https` it doesn't work as expected.

The configuration looks like that:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/9/b9c04ec8379f2e053f56730b260c61f2a5d2eb32.png)

for example let's suppose the link from where we download packetbeat is a malware:

```auto
https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-7.10.2-windows-x86_64.zip

```

The output from my logstash filter will look like this:

```auto
uri_proto: https
domain: artifacts.elastic.co
uri_para: downloads/beats/packetbeat/packetbeat-7.10.2-windows-x86_64.zip
url: https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-7.10.2-windows-x86_64.zip

```

But when I click on the link in my windows machine where packetbeat installed, all what I can get in the logs of packetbeat is : `artifacts.elastic.co`, the rest of the `url` is not captured neither on `type: https` nor on `type:dns`.

and if I use just the field `artifacts.elastic.co`, it will generate a lot of false positive,as like in this example we suggested only the link to packetbeat download which is a malware and not all what is in the `artifacts.elastic.co`

till now I didn't find a solution for that problem, Hope you did undestand what I explained and you can give me an idea.

Best regards

---

<div class="post-metadata">

**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\
**Post date:** [January 19, 2021, 2:00pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/15 "2021-01-19T14:00:12Z")

</div>

Hi @TheHunter1, Glad to see you have your rule running!

I am sorry if I am missing the question, but is this the same issue we discussed earlier?

> [@Mike\_Paquette](#):
>
> `url.full` is not populated in DNS or TLS events from Packetbeat, because the URL is either not present or not visible in those protocols.

In the case of TLS, the full URL is not available to Packetbeat, since only the domain (not the full URL) is visible outside of the encrypted payload.

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 19, 2021, 2:12pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/16 "2021-01-19T14:12:38Z")

</div>

So if I understand well, there is no way to create a rule to track malicious URLs if it's using https, right ?

---

<div class="post-metadata">

**Author:** ![Mike\_Paquette](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mike_paquette/32/119011_2.png) [@Mike\_Paquette](https://discuss.elastic.co/u/Mike_Paquette)\
**Post date:** [January 19, 2021, 2:46pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/17 "2021-01-19T14:46:51Z")

</div>

> So if I understand well, there is no way to create a rule to track malicious URLs if it's using https, right ?

Yes, that's correct for Packetbeat and other passive network monitoring /data collection systems.

---

<div class="post-metadata">

**Author:** ![TheHunter1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thehunter1/32/80190_2.png) [@TheHunter1](https://discuss.elastic.co/u/TheHunter1)\
**Post date:** [January 19, 2021, 2:49pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/18 "2021-01-19T14:49:23Z")

</div>

Thanks for all your answers 😊

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 16, 2021, 2:49pm UTC](https://discuss.elastic.co/t/urlhaus-threat-intelligence-create-a-new-rule/260857/19 "2021-02-16T14:49:23Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.