How to drop DNS event if they are present into top 1 million file

[yquirion]

Dear all,

I'm wonderion how to configure a logstash pipeline that will handle my DNS logs. From that logs, there are lots of logs I don't want to see because I know they are legitimate.

So I would like to have logstash looking into the Top One Million file (Cisco Popularity List) and drop all domains that are matching this file.

The file has the following format:

1,google.com
2,www.google.com
3,microsoft.com
4,netflix.com
5,data.microsoft.com
6,cloud.netflix.com
7,prod.cloud.netflix.com
8,ftl.netflix.com
9,prod.ftl.netflix.com
10,nrdp.prod.cloud.netflix.com

The file is pretty huge, and I would like to have it very performant. Our DNS is sending more than 1k events by seconds, so it needs to be very powerfull.

Does someone has an idea how to configure this with logstash?

Thank you and best regards,
Yanick

[Badger]

I would reorder the columns so that the domain name comes first, then use a translate filter to do the lookup, and drop {} the event if it gets a match.

[yquirion]

Hi Badger!

Thank you very much for your answer.

I look into the translate plugin, but I can't see any "drop" function to be used with the translate.

Should I, for example, use add_tag to add a specific tag on sucessfull match then use drop{} when that specific tag is present to drop the event?

Thanks!
Yanick

[Badger]

You can use the target option to tell the translate where to write the looked up value

target => "[@metadata][lookup]"

then test whether it exists

if [@metadata][lookup] { drop {} }

Doing it using the add_tag option would likely also work.

[yquirion]

Hi Badger,

It is working fine. The question I have, do you think that dictinnary file, who contains 1M lines can slow down my logstash servers? So far, we only add the secondary DNS server and everything is responding just fine, but when I will add the primary DNS server, this one has 3-4 time more requests by seconds.

Thank you again for your help. I was looking to someting different before you siggest me the translate plugin. I tought there was other plugins that can read CSV file.

Regards,
Yanick

[Badger]

The dictionary that translate uses is basically a Ruby hash, and the performance of that should not significantly decline as the number of entries increases.