# Compare two datasets

**URL:** <https://discuss.elastic.co/t/compare-two-datasets/129006>\
**Category:** Logstash\
**Created:** [April 22, 2018, 1:27pm UTC](https://discuss.elastic.co/t/compare-two-datasets/129006 "2018-04-22T13:27:44Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![counterf](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@counterf](https://discuss.elastic.co/u/counterf)\
**Post date:** [April 22, 2018, 1:27pm UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/1 "2018-04-22T13:27:45Z")

</div>

Hello,

I have a list containing millions of malicious IP addresses, and I am using ELK to get and analyze data from multiple firewalls.

I received millions of logs per days from the firewall and my malicious IP address list is changing constantly.

I want to create a search or something else that compares the firewall logs and this malicious list of IP addresses.  
If there is any match, the search should return the results, create an alarm, whatever. I just want to be aware of any match.

Can someone point me a direction ?  
I have been exhausting my neuron on this task.

---

<div class="post-metadata">

**Author:** ![Kumar\_Narottam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kumar_narottam/32/18959_2.png) [@Kumar\_Narottam](https://discuss.elastic.co/u/Kumar_Narottam)\
**Post date:** [April 22, 2018, 7:31pm UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/2 "2018-04-22T19:31:42Z")

</div>

Hi Jack,

An IP Address is made of 4 parts.  
A.B.C.D

You need to anlayse this IP address in multi field search.

1. The first analyser can be a stop analyser where you can separate IP address on basis of . (dot).
2. The second analyser can be a standard analyser which a char filter which removes . out of IP address and makes the entire IP address as single string.

So when you run your analysis you can Multi search your new IP address with both analysed form of text we have. Any score of beyond 80 percent can be considered as malicious IP for you. The percentage can be tweaked as per use case.

Let me know if this works for you.

---

<div class="post-metadata">

**Author:** ![counterf](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@counterf](https://discuss.elastic.co/u/counterf)\
**Post date:** [April 22, 2018, 7:49pm UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/3 "2018-04-22T19:49:24Z")

</div>

Actually, I am following another approach...  
I was trying to compare two indices in elastic, and I was consuming a lot of time not getting anywhere.

I think the best way to compare data from the malicius hosts and the logs from the fws is to create a lookup/translate as soon as we get logs and enrich the incoming log with additional information.  
Basically, my plan is to do a lookup/translate on the src/dst IP. If there is a match, I will tag the src/dst IP as malicious.

Here is the logstash filter I am preparing:

> ```
> filter {
> translate {
> field => "source_ip" //source IP from the original log. 
> destination => "malicious" //new field created to tag the IP as malicious
> dictionary_path => '/opt/minemeld/ipv4.yaml' //list of Malicious addresses in YAML format
> refresh_interval => '300' //refresh interval for YAML file
> }
> }
> 
> ```

The YAML file should look like this:

> “1.1.1.1”:”TRUE”  
> “2.2.2.2”:”TRUE”  
> …

I cant test today, but I believe it will work

Here is more info if for those who face the same issue

> **[Enriching Data with Lookups | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/lookup-enrichment.html)**

> **[Translate filter plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html#plugins-filters-translate-dictionary_path)**

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [April 22, 2018, 8:55pm UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/4 "2018-04-22T20:55:29Z")

</div>

That's the best way, that or use an Elasticsearch index with the data in it and then an Elasticsearch filter that does a lookup to that index.

---

<div class="post-metadata">

**Author:** ![counterf](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@counterf](https://discuss.elastic.co/u/counterf)\
**Post date:** [April 23, 2018, 12:02am UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/5 "2018-04-23T00:02:39Z")

</div>

The translate is not working.

I am using packetbeat to send network activity.

Here is my conf:

> # input section
> 
> input {  
> beats {  
> port =\> 5044  
> }  
> }
> 
> filter {  
> translate {  
> field =\> "dest.ip"  
> destination =\> "malicious\_IP"  
> dictionary\_path =\> '/opt/logstash/maliciousIPV4.yaml'  
> override =\> true  
> }
> 
> translate {  
> field =\> "source.ip"  
> destination =\> "malicious\_IP"  
> dictionary\_path =\> '/opt/logstash/maliciousIPV4.yaml'  
> override =\> true  
> }
> 
> }  
> output {  
> elasticsearch {  
> hosts =\> localhost  
> index =\> "%{[@metadata][beat]}-%{+YYYY.MM.dd}"  
> }  
> stdout {  
> codec =\> rubydebug  
> }  
> }

I can see the logs coming, they appear in Kibana, but the translation just doesnt work.

Here is the content of the dictionary

> "216.46.173.126": "true"  
> "180.179.174.219": "true"  
> "204.77.168.241": "true"  
> "65.39.197.164": "true"  
> "80.91.33.133": "true"  
> "84.208.15.12": "true"  
> "74.125.60.158": "true"  
> "8.8.8.8": "true"  
> "200.221.2.45": "true"  
> "186.232.248.40": "true"

The translate plugin is installed.

Any idea?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [April 23, 2018, 6:17am UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/6 "2018-04-23T06:17:51Z")

</div>

> [@counterf](#):
>
> field =\> "source.ip"

I think this should be: `field => "[source][ip]"`

---

<div class="post-metadata">

**Author:** ![counterf](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@counterf](https://discuss.elastic.co/u/counterf)\
**Post date:** [April 23, 2018, 11:40am UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/7 "2018-04-23T11:40:52Z")

</div>

it worked!.

Thanks a lot. I spent 4 hours yesterday trying to fix this lol.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/5/4534a8f9324ddc30105568dc1b839ca70c61c160.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/6/86e42fcc55c317eb834cedefe9c63c2719d2f526.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9684873880de98b1f1f6ab89a331b7ba05f1577.png)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 21, 2018, 11:40am UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/8 "2018-05-21T11:40:54Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.