# Compare two datasets

**URL:** <https://discuss.elastic.co/t/compare-two-datasets/129006>\
**Category:** Logstash\
**Created:** [April 22, 2018, 1:27pm UTC](https://discuss.elastic.co/t/compare-two-datasets/129006 "2018-04-22T13:27:44Z")\
**Posts on this page:** 1\
**Showing post:** 3

<div class="post-metadata">

**Author:** ![counterf](https://avatars.discourse-cdn.com/v4/letter/c/9d8465/32.png) [@counterf](https://discuss.elastic.co/u/counterf)\
**Post date:** [April 22, 2018, 7:49pm UTC](https://discuss.elastic.co/t/compare-two-datasets/129006/3 "2018-04-22T19:49:24Z")

</div>

Actually, I am following another approach...  
I was trying to compare two indices in elastic, and I was consuming a lot of time not getting anywhere.

I think the best way to compare data from the malicius hosts and the logs from the fws is to create a lookup/translate as soon as we get logs and enrich the incoming log with additional information.  
Basically, my plan is to do a lookup/translate on the src/dst IP. If there is a match, I will tag the src/dst IP as malicious.

Here is the logstash filter I am preparing:

> ```
> filter {
> translate {
> field => "source_ip" //source IP from the original log. 
> destination => "malicious" //new field created to tag the IP as malicious
> dictionary_path => '/opt/minemeld/ipv4.yaml' //list of Malicious addresses in YAML format
> refresh_interval => '300' //refresh interval for YAML file
> }
> }
> 
> ```

The YAML file should look like this:

> “1.1.1.1”:”TRUE”  
> “2.2.2.2”:”TRUE”  
> …

I cant test today, but I believe it will work

Here is more info if for those who face the same issue

> **[Enriching Data with Lookups | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/lookup-enrichment.html)**

> **[Translate filter plugin | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/plugins-filters-translate.html#plugins-filters-translate-dictionary_path)**

---

_[View the full topic](https://discuss.elastic.co/t/compare-two-datasets/129006)._