# Logstash Data Enrichment with URLHAUS

**URL:** <https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756>\
**Category:** Logstash\
**Created:** [October 2, 2018, 7:48pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756 "2018-10-02T19:48:48Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![martb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martb/32/78185_2.png) [@martb](https://discuss.elastic.co/u/martb)\
**Post date:** [October 2, 2018, 7:48pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/1 "2018-10-02T19:48:48Z")

</div>

Hi,

After watching the recent elastic threat hunting Webex which mentioned URLHAUS, so I wanted to try to add a lookup to the haus csv data source.

Currently in my logstash conf Im connecting to a database which contains a urlpath field, I then want to check this against the URLs contained in the haus list(cvs). Ultimately return true or false and pull back corresponding info from the csv if matched ie related tags etc

I had a few unsuccessful attempts and wondered if anyone has some tips or examples to follow.

Any help would be much appreciated.

Regards

Martin

---

<div class="post-metadata">

**Author:** ![bardie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bardie/32/36173_2.png) [@bardie](https://discuss.elastic.co/u/bardie)\
**Post date:** [October 3, 2018, 1:38pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/2 "2018-10-03T13:38:26Z")

</div>

> [@martb](#):
>
> m connecting to a database which contains a urlpath field, I then want to check this against the URLs contained in the haus list(cvs). Ultimately return true or false and pull back corresponding info from the csv if matched ie related tags etc
> 
> I had a few unsuccessful attempts and wondered if anyone has some tips or examples to f

Use the following config to query URLHAUS once a day (you can change the interval) and output to elasticsearch index - malware

```
input {
  exec {
    command => 'curl https://urlhaus.abuse.ch/downloads/csv/'
    interval => 86400
    type => 'iphaus'
    codec => line
  } 
}
filter {
  if [type] == "iphaus" {
    csv {
      columns => ["id","dateadded","url","url_status","threat","tags","urlhaus_link"]
      separator => ","
    }
  }
} 

output {
 stdout { codec => json }
     elasticsearch {
        hosts => ["elasticsearch"]
        index => "malware-%{+YYYY.MM.dd}"
     }
   }

```

Add this filter to your config to compare your %{[url]} to the blacklisted urls stored in elasticsearch

```
filter {
       elasticsearch {
       hosts => ["elasticsearch"]
       index => "malware-*"
       query => 'url:"%{[url]}"'
       fields => { "link" => "reference" }
     } 
}

```

---

<div class="post-metadata">

**Author:** ![martb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martb/32/78185_2.png) [@martb](https://discuss.elastic.co/u/martb)\
**Post date:** [October 3, 2018, 2:12pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/3 "2018-10-03T14:12:44Z")

</div>

thanks bardie for the reply this is really helpful.

I've successfully pulled the data from haus and created an index called malware-\* but I'm not clear on the second part i.e  
filter {  
elasticsearch {  
hosts =\> ["elasticsearch"]  
index =\> "malware-\*"  
query =\> 'url:"%{[url]}"'  
fields =\> { "link" =\> "reference" }  
}  
}

Is this added to my existing conf that contain my urlpath field which I want to compare?  
This is the conf output that populates my existing index which contains urlpath I want to match against:

output {  
if [type] == "test-2-3" {  
elasticsearch {  
hosts =\> "[http://xxx.xxx.xxx.xxx:9200](http://xxx.xxx.xxx.xxx:9200)"  
user =\> "ELASTIC\_USER"  
password =\> "ELASTIC\_PWD"  
index =\> "test-main-controls-%{+YYYY.MM.dd}"  
}  
}  
}

I've not had any exposure to using this compare\match method so would be great to get this working, thanks again

Also can I use regex in the query to check any part of the url for a match?

---

<div class="post-metadata">

**Author:** ![bardie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bardie/32/36173_2.png) [@bardie](https://discuss.elastic.co/u/bardie)\
**Post date:** [October 3, 2018, 2:42pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/4 "2018-10-03T14:42:00Z")

</div>

Logstash has a filter plugin called elasticsearch which helps you query elasticsearch and extract data.

To install the elasticsearch filter plugin run

```
/usr/share/logstash/bin/logstash-plugin logstash-filter-elasticsearch

```

After installation, add the following section to the logstash config file that you want to compare.

```
  elasticsearch {
     hosts => ["elasticsearch"]
     index => "malware-*"
     query => 'url:"%{[url]}"'
     fields => { "link" => "reference" }
  }

```

Make sure that your parser has a field called url (`query => 'url:"%{[url]}"'`) so that the query to elasticsearch will work.

To understand the function of the plugin. Please reference the following link:  
[https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html)

---

<div class="post-metadata">

**Author:** ![martb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martb/32/78185_2.png) [@martb](https://discuss.elastic.co/u/martb)\
**Post date:** [October 3, 2018, 3:14pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/5 "2018-10-03T15:14:07Z")

</div>

Hi,

thanks. I now have it running. One last question regarding : query =\> 'url:"%{[url]}"'  
So I'm clearly can I change "%{[url]}"' to "%{[urlpath]}" or do they have to match exactly?

---

<div class="post-metadata">

**Author:** ![bardie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bardie/32/36173_2.png) [@bardie](https://discuss.elastic.co/u/bardie)\
**Post date:** [October 3, 2018, 3:15pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/6 "2018-10-03T15:15:05Z")

</div>

Yes, you can change it. It was just an example

---

<div class="post-metadata">

**Author:** ![martb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/martb/32/78185_2.png) [@martb](https://discuss.elastic.co/u/martb)\
**Post date:** [October 3, 2018, 3:17pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/7 "2018-10-03T15:17:34Z")

</div>

many thanks, much appreciated.

---

<div class="post-metadata">

**Author:** ![bardie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bardie/32/36173_2.png) [@bardie](https://discuss.elastic.co/u/bardie)\
**Post date:** [October 3, 2018, 3:18pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/8 "2018-10-03T15:18:25Z")

</div>

your welcome 👍

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 31, 2018, 3:18pm UTC](https://discuss.elastic.co/t/logstash-data-enrichment-with-urlhaus/150756/9 "2018-10-31T15:18:30Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.