How effective translate field in logstash

Elastic Stack Logstash
1

Hi,

I have a set of threat intel feeds i want to check whether those blacklisted ip occurs in my logs... i tried using translate field it works fine...i want to know whether we can compare billions of blacklisted ip using translate field or is there any limitation

Thanks

2

With a translate filter it keeps a hash of all the entries in memory, so having billions of entries will require a large amount of memory, but if you have the memory it should be OK.

A cidr filter might be a better fit, depending on the details on the set of IPs.

3

ok thanks.... but again cidr filter uses only three fields a particular address,network,network path so as per my understanding i should define all the ip in a file and use the network_path field which points to that file ...

filter {
cidr {
add_tag => [ "linklocal" ]
address => [ "%{clientip}" ]
network_path => [ "/opt/blacklistedip" ]
}
}

4

and one more doubt is there any other method to combine threat intel and elastic