Logstash Data Enrichment with URLHAUS

Elastic Stack Logstash
1

Hi,

After watching the recent elastic threat hunting Webex which mentioned URLHAUS, so I wanted to try to add a lookup to the haus csv data source.

Currently in my logstash conf Im connecting to a database which contains a urlpath field, I then want to check this against the URLs contained in the haus list(cvs). Ultimately return true or false and pull back corresponding info from the csv if matched ie related tags etc

I had a few unsuccessful attempts and wondered if anyone has some tips or examples to follow.

Any help would be much appreciated.

Regards

Martin

2

Use the following config to query URLHAUS once a day (you can change the interval) and output to elasticsearch index - malware

input {
  exec {
    command => 'curl https://urlhaus.abuse.ch/downloads/csv/'
    interval => 86400
    type => 'iphaus'
    codec => line
  } 
}
filter {
  if [type] == "iphaus" {
    csv {
      columns => ["id","dateadded","url","url_status","threat","tags","urlhaus_link"]
      separator => ","
    }
  }
} 

output {
 stdout { codec => json }
     elasticsearch {
        hosts => ["elasticsearch"]
        index => "malware-%{+YYYY.MM.dd}"
     }
   }

Add this filter to your config to compare your %{[url]} to the blacklisted urls stored in elasticsearch

filter {
       elasticsearch {
       hosts => ["elasticsearch"]
       index => "malware-*"
       query => 'url:"%{[url]}"'
       fields => { "link" => "reference" }
     } 
}
3

thanks bardie for the reply this is really helpful.

I've successfully pulled the data from haus and created an index called malware-* but I'm not clear on the second part i.e
filter {
elasticsearch {
hosts => ["elasticsearch"]
index => "malware-*"
query => 'url:"%{[url]}"'
fields => { "link" => "reference" }
}
}

Is this added to my existing conf that contain my urlpath field which I want to compare?
This is the conf output that populates my existing index which contains urlpath I want to match against:

output {
if [type] == "test-2-3" {
elasticsearch {
hosts => "http://xxx.xxx.xxx.xxx:9200"
user => "ELASTIC_USER"
password => "ELASTIC_PWD"
index => "test-main-controls-%{+YYYY.MM.dd}"
}
}
}

I've not had any exposure to using this compare\match method so would be great to get this working, thanks again

Also can I use regex in the query to check any part of the url for a match?

4

Logstash has a filter plugin called elasticsearch which helps you query elasticsearch and extract data.

To install the elasticsearch filter plugin run

/usr/share/logstash/bin/logstash-plugin logstash-filter-elasticsearch

After installation, add the following section to the logstash config file that you want to compare.

  elasticsearch {
     hosts => ["elasticsearch"]
     index => "malware-*"
     query => 'url:"%{[url]}"'
     fields => { "link" => "reference" }
  }

Make sure that your parser has a field called url (query => 'url:"%{[url]}"') so that the query to elasticsearch will work.

To understand the function of the plugin. Please reference the following link:
https://www.elastic.co/guide/en/logstash/current/plugins-filters-elasticsearch.html

1 Like
5

Hi,

thanks. I now have it running. One last question regarding : query => 'url:"%{[url]}"'
So I'm clearly can I change "%{[url]}"' to "%{[urlpath]}" or do they have to match exactly?

6

Yes, you can change it. It was just an example

7

many thanks, much appreciated.

8

your welcome :+1:

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.