[ URLHaus threat intelligence ]: create a new rule

[TheHunter1]

Hello,

I am using logsatsh to enrich my SIEM with malicious Urls that I downloaded from URLHaus,
and I wanna create a rule that send an alert everytime a user connect to one of the malicious urls.
The problem is that I didn't find a field in packetbeat which contains the full url, so I couldn't use the Match alert

for example:
Malicious url:

http://www.davidephoto.it/GsnIO/

and packetbeat fields:

**dns**
dns.question.name: www.davidephoto.it
dns.question.etld_plus_one: davidephoto.it
dns.question.registered_domain: davidephoto.it
dns.question.subdomain: www
dns.question.top_level_domain: it

**tls**
destination.domain: www.davidephoto.it
server.domain: www.davidephoto.it
tls.client.server_name: www.davidephoto.it

Could you tell me if I can use something like contain as non of the fields Match completly to the url ? or any other solution

Thanks for your help

[warkolm (Mark Walkom)]

Is this specifically in the SIEM app?

[TheHunter1]

it's for security reaspns, to detect malicious connections, so yes it's for SIEM utilisation

[warkolm (Mark Walkom)]

Ok, if it's in the SIEM app though I will move the topic to that category.

[Mike_Paquette]

Hi @TheHunter1,

Elastic Common Schema[1] defines a field for storing the complete URL as url.full, as documented here.

Packetbeat populates the url.full field for http events (i.e., where network.protocol: "http").

url.full is not populated in DNS or TLS events from Packetbeat, because the URL is either not present or not visible in those protocols.

Perhaps instead of using a "URL" threat list, you might consider a "DNS" or "Domain"-based threat list, which you will be able to match against your DNS events. You could then compare the dns.question.name field in your events against the values in this list.

Here's an interesting example of such as list (https://feeds.alphasoc.net/ryuk.txt) that you might want to investigate.

[1]
I want to ensure that you are aware of Elastic Common Schema (ECS).

The Elastic SIEM/Security app, including its detection rules, signals, and detection alerts, requires your data to be indexed in an ECS-compliant format. ECS is an open source, community-developed schema that specifies field names and Elasticsearch data types for each field, and provides descriptions and example usage.

The easiest way to get your data in ECS-compliant format is to use an Elastic-supplied beat module, (e.g., filebeat or Elastic Agent integration), which will ingest and index your data in an ECS-compliant format. Elastic provides a growing list of these integrations that you can find on our Integrations page.

If you're using a custom data ingestion method (like your use of Logstash), or one provided by a third-party, then you may need to convert your data so that it is in an ECS-compliant format before you can use the SIEM/security app. This can be done by creating your own beat/module, or your own Logstash configuration for each data source, which will convert your data to ECS during the ingestion process.

General guidelines for creating ECS-compliant data:

1. Each indexed document (e.g., your log, event, etc.) MUST have the @timestamp field.
2. Your index mapping template must specify the Elasticsearch field data type for each field as defined by ECS. For example, your @timestamp field must be of the date field data type, etc.. This ensures that there will not be any mapping conflicts in your indices.
3. The original fields from your log/event SHOULD be copied/renamed/converted to the corresponding ECS-defined field name and data type.
4. Additional ECS fields, such as the ECS Categorization fields SHOULD be populated for each log/event, to allow proper inclusion of your data into dashboards and detection rules.

A list of the specific ECS fields used by the SIEM/Security app is provided in this reference.

[TheHunter1]

Thanks for all your exaplanation @Mike_Paquette, it was so clair
so as I can understand I can't use the rule Match in this case as there is no field that contains the full URL in DNS and TLS
is there a posibility to create a rule with something like IN to match just some of the full URL
Exemple:

The full URL is:

http://www.davidephoto.it/GsnIO/

and we can see in DNS dns.question.name and TLS destination.domain: they both contain the url www.davidephoto.it

The idea is to say if: dns.question.name OR destination.domain: Contain full.url then alerte

we can see that www.davidephoto.it IS IN http://www.davidephoto.it/GsnIO/

Hope you can understand what I am meaninig

[hilo21 (Hamza Ouadiâ)]

Hello,

Can you tell me please what kind of alerting mechanism you're using ?

• Watcher
• SIEM Detections

Also is the URLhaus events stored in a different index than your packetbeat or are these fields on the same Document ?

Thank you

[TheHunter1]

Thanks for your answer @hilo21,

So I want if possibile to create a rule from the SIEM detections without using a script to create a watcher, unless if there is no way to do it with the SIEM rules.

and for your second question, I configured logsatsh to download the URLHaus database each 24h and store it in a dedicated index just for URLHaus.

[Mike_Paquette]

Hi @TheHunter1 To further clarify your question, I think you are asking if a SIEM/Security App detection rule can MATCH a keyword field such as dns.question.name or destination.domain when the keyword field value is contained in the URL field. Like this (shown with the Security App Indicator Match rule type):

Is that your question?

[TheHunter1]

Thanks for your answer @Mike_Paquette,
But by doing that I think that it won't work as there is not a 100% MATCH between the 2 urls.

The threat url: http://www.davidephoto.it/GsnIO/
dns.question.name in packetbeat field: www.davidephoto.it

I think theMatch will work only if there is a 100% match between the 2 fields, isn't it ?

[hilo21 (Hamza Ouadiâ)]

One way of manh to do this is to use elasticsearch filter plugin in packetbeat logstash pipeline to send a DSL query to elasticsearch for whatever index you want. I guess whith some testing you would be able to do it. Try wildcard query maybe it will work. Never used it in this use case but you can try this workaround.
But what you should be doing is parsing the urls to get site, etld and tld out of them this way your data would be a little bit cleaner and flexible for many use cases.

[Mike_Paquette]

Thanks @TheHunter1 I was just trying to understand your question

You are correct that the MATCH implemented in the Indicator Match rule type is an exact match, and will NOT address your specific request.

For now, you'll need to have separate indicator match rules for each type of threat indicator you want to compare against.

Please let us know how you decide to proceed. Thanks!

[TheHunter1]

Hello again,

So I did as you and @hilo21 suggested, I used agrok filter, and separated the indicators.
The grok filter I used is:

    grok {
      match => { "url" => "%{URIPROTO:uri_proto}:\/\/(%{IP:destination.ip}:%{NUMBER:destination.port})?(%{URIHOST:domain})?\/%{GREEDYDATA:uri_para}" }
    }

( somtimes the indicator is an IP address and port and sometimes it's URL, that why my filter looks like that)

and my rule is like that:

the only problem is that I am getting this error while creating the rule so I couldn't test if it's working correctly :

So I created a new Topic here :

[ Creating new rule ]: ERROR Authentication using apikey failed - api key has been invalidated

Hello, I am using elasticsearch, kibana and beats version 7.10.1 with a Trial license, I am trying to create a new Match rule and in my elasticsearch node I am getting this error: [2021-01-18T11:33:18,032][WARN ][o.e.x.s.a.AuthenticationService] [DATA-03] Authentication using apikey failed - api key has been invalidated [2021-01-18T11:33:18,035][WARN ][o.e.x.s.a.AuthenticationService] [DATA-03] Authentication using apikey failed - api key has been invalidated and in kibana interface: Rule …

[TheHunter1]

Hello again @Mike_Paquette, @hilo21 ,
So the problem of the creation of the rule has been solved,

for the http URL It's working fine as I am using the url.full of packetbeat, but for https it doesn't work as expected.

The configuration looks like that:

for example let's suppose the link from where we download packetbeat is a malware:

https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-7.10.2-windows-x86_64.zip

The output from my logstash filter will look like this:

uri_proto: https
domain: artifacts.elastic.co
uri_para: downloads/beats/packetbeat/packetbeat-7.10.2-windows-x86_64.zip
url: https://artifacts.elastic.co/downloads/beats/packetbeat/packetbeat-7.10.2-windows-x86_64.zip

But when I click on the link in my windows machine where packetbeat installed, all what I can get in the logs of packetbeat is : artifacts.elastic.co, the rest of the url is not captured neither on type: https nor on type:dns.

and if I use just the field artifacts.elastic.co, it will generate a lot of false positive,as like in this example we suggested only the link to packetbeat download which is a malware and not all what is in the artifacts.elastic.co

till now I didn't find a solution for that problem, Hope you did undestand what I explained and you can give me an idea.

Best regards

[Mike_Paquette]

Hi @TheHunter1, Glad to see you have your rule running!

I am sorry if I am missing the question, but is this the same issue we discussed earlier?

@Mike_Paquette

url.full is not populated in DNS or TLS events from Packetbeat, because the URL is either not present or not visible in those protocols.

In the case of TLS, the full URL is not available to Packetbeat, since only the domain (not the full URL) is visible outside of the encrypted payload.

[TheHunter1]

So if I understand well, there is no way to create a rule to track malicious URLs if it's using https, right ?

[Mike_Paquette]

So if I understand well, there is no way to create a rule to track malicious URLs if it's using https, right ?

Yes, that's correct for Packetbeat and other passive network monitoring /data collection systems.

[TheHunter1]

Thanks for all your answers