Compare two datasets

Elastic Stack Logstash
3

Actually, I am following another approach...
I was trying to compare two indices in elastic, and I was consuming a lot of time not getting anywhere.

I think the best way to compare data from the malicius hosts and the logs from the fws is to create a lookup/translate as soon as we get logs and enrich the incoming log with additional information.
Basically, my plan is to do a lookup/translate on the src/dst IP. If there is a match, I will tag the src/dst IP as malicious.

Here is the logstash filter I am preparing:

filter {
  translate {
    field => "source_ip"  //source IP from the original log. 
    destination => "malicious"  //new field created to tag the IP as malicious
    dictionary_path => '/opt/minemeld/ipv4.yaml'  //list of Malicious addresses in YAML format
    refresh_interval => '300'   //refresh interval for YAML file
  }
}

The YAML file should look like this:

“1.1.1.1”:”TRUE”
“2.2.2.2”:”TRUE”
…

I cant test today, but I believe it will work

Here is more info if for those who face the same issue

1 Like