# Compare fields with an external file

**URL:** <https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492>\
**Category:** Logstash\
**Created:** [March 15, 2019, 9:08am UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492 "2019-03-15T09:08:10Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\
**Post date:** [March 15, 2019, 9:08am UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/1 "2019-03-15T09:08:10Z")

</div>

Hi everybody,

I'm dealing with a problem since a few days.

First, I'm unable to delete the field message in kibana. I try with mutate, or directly in the grok filter, but there's always the field in kibana after a refresh.

Second point, I have a multiple grok filter, and the two first are closely equals, but there's a little difference : there's one with more fields (the first one have xlatedport xlatesport and xlatesrc) for example) and the second has not. In the current state, it only matches the first, the third and the 4th filter.

(To read it easily, I made some space after the comas. There's no space in my config file).

Thanks a lot.

filter {  
grok {  
match =\> { "message" =\> ["%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}\<s\_port\>%{NUMBER:source-port}\</s\_port\>\<d\_port\>%{NUMBER:destination-port}\</d\_port\>%{WORD:action}%{NUMBER: **xlatedport** }%{NUMBER: **xlatesport** }%{IP: **xlatesrc** }%{WORD:service}\<i\_f\_dir\>%{WORD:iface-direction}\</i\_f\_dir\>\<i\_f\_name\>%{WORD:iface-name}\</i\_f\_name\>%{NUMBER:rule-id}%{GREEDYDATA:product}\<state%{GREEDYDATA:state}/\>",

"%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}\<s\_port\>%{NUMBER:source-port}\</s\_port\>\<d\_port\>%{NUMBER:destination-port}\</d\_port\>%{WORD:action}%{WORD:service}\<i\_f\_dir\>%{WORD:iface-direction}\</i\_f\_dir\>\<i\_f\_name\>%{GREEDYDATA:iface-name}\</i\_f\_name\>%{NUMBER:rule-id}%{GREEDYDATA:product}\<state%{GREEDYDATA:state}/\>",

"1 %{GREEDYDATA:timestamp} %{GREEDYDATA:equipement} RT\_FLOW - [%{GREEDYDATA:session} source-address="%{IP:origin-ip}" source-port="%{NUMBER:source-port}" destination-address="%{IP:dst-ip}" destination-port="%{NUMBER:destination-port}" service-name="%{GREEDYDATA:service-name}" nat-source-address="%{IP:nat-source-address}" nat-source-port="%{NUMBER:nat-source-port}" nat-destination-address="%{IP:nat-destination-address}" nat-destination-port="%{NUMBER:nat-destination-port}" src-nat-rule-type="%{GREEDYDATA:src-nat-rule-type}" src-nat-rule-name="%{GREEDYDATA:src-nat-rule-name}" dst-nat-rule-type="%{GREEDYDATA:dst-nat-rule-type}" dst-nat-rule-name="%{GREEDYDATA:dst-nat-rule-name}" protocol-id="%{NUMBER:protocol-id}" policy-name="%{GREEDYDATA:policy-name}" source-zone-name="%{WORD:source-zone-name}" destination-zone-name="%{WORD:destination-zone-name}" session-id-32="%{GREEDYDATA:session-id-32}" username="%{GREEDYDATA:username}" roles="%{GREEDYDATA:roles}" packet-incoming-interface="%{GREEDYDATA:packet-incoming-interface}" application="%{GREEDYDATA:application}" nested-application="%{GREEDYDATA:nested-application}" encrypted="%{GREEDYDATA:encrypted}"]",

"%{GREEDYDATA:firewall}: NetScreen device\_id=%{GREEDYDATA:device} [Root]system-notification-00257(traffic): start\_time="%{GREEDYDATA:timestamp}" duration=%{NUMBER:duration} policy\_id=%{NUMBER:policy\_id} service=%{WORD:service} proto=%{NUMBER:proto} src zone=%{WORD:src-zone} dst zone=%{WORD:dst-zone} action=%{WORD:action} sent=%{NUMBER:sent} rcvd=%{NUMBER:rcvd} src=%{IP:src-ip} dst=%{IP:dst-ip} src\_port=%{NUMBER:source-port} dst\_port=%{NUMBER:destination-port} session\_id=%{NUMBER:session-id} reason=%{GREEDYDATA:reason}"]}  
break\_on\_match =\> "false"  
remove\_field =\> ["message"]  
}

}

Edit 🙂

It seems I find the right solution.

filter {  
grok {  
match =\> { "message" =\> ["%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}\<s\_port\>%{NUMBER:source-port}\</s\_port\>\<d\_port\>%{NUMBER:destination-port}\</d\_port\>%{WORD:action}%{NUMBER:xlatedport}%{NUMBER:xlatesport}%{IP:xlatesrc}%{WORD:service}\<i\_f\_dir\>%{WORD:iface-direction}\</i\_f\_dir\>\<i\_f\_name\>%{WORD:iface-name}\</i\_f\_name\>%{NUMBER:rule-id}%{GREEDYDATA:product}\<state%{GREEDYDATA:state}/\>",

"%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}\<s\_port\>%{NUMBER:source-port}\</s\_port\>\<d\_port\>%{NUMBER:destination-port}\</d\_port\>%{WORD:action}%{WORD:service}\<i\_f\_dir\>%{WORD:iface-direction}\</i\_f\_dir\>\<i\_f\_name\>%{GREEDYDATA:iface-name}\</i\_f\_name\>%{NUMBER:rule-id}%{GREEDYDATA:product}\<state%{GREEDYDATA:state}/\>",

"1 %{GREEDYDATA:timestamp} %{GREEDYDATA:equipement} RT\_FLOW - [%{GREEDYDATA:session} source-address="%{IP:origin-ip}" source-port="%{NUMBER:source-port}" destination-address="%{IP:dst-ip}" destination-port="%{NUMBER:destination-port}" service-name="%{GREEDYDATA:service-name}" nat-source-address="%{IP:nat-source-address}" nat-source-port="%{NUMBER:nat-source-port}" nat-destination-address="%{IP:nat-destination-address}" nat-destination-port="%{NUMBER:nat-destination-port}" src-nat-rule-type="%{GREEDYDATA:src-nat-rule-type}" src-nat-rule-name="%{GREEDYDATA:src-nat-rule-name}" dst-nat-rule-type="%{GREEDYDATA:dst-nat-rule-type}" dst-nat-rule-name="%{GREEDYDATA:dst-nat-rule-name}" protocol-id="%{NUMBER:protocol-id}" policy-name="%{GREEDYDATA:policy-name}" source-zone-name="%{WORD:source-zone-name}" destination-zone-name="%{WORD:destination-zone-name}" session-id-32="%{GREEDYDATA:session-id-32}" username="%{GREEDYDATA:username}" roles="%{GREEDYDATA:roles}" packet-incoming-interface="%{GREEDYDATA:packet-incoming-interface}" application="%{GREEDYDATA:application}" nested-application="%{GREEDYDATA:nested-application}" encrypted="%{GREEDYDATA:encrypted}"]",

"%{GREEDYDATA:firewall}: NetScreen device\_id=%{GREEDYDATA:device} [Root]system-notification-00257(traffic): start\_time="%{GREEDYDATA:timestamp}" duration=%{NUMBER:duration} policy\_id=%{NUMBER:policy\_id} service=%{WORD:service} proto=%{NUMBER:proto} src zone=%{WORD:src-zone} dst zone=%{WORD:dst-zone} action=%{WORD:action} sent=%{NUMBER:sent} rcvd=%{NUMBER:rcvd} src=%{IP:src-ip} dst=%{IP:dst-ip} src\_port=%{NUMBER:source-port} dst\_port=%{NUMBER:destination-port} session\_id=%{NUMBER:session-id} reason=%{GREEDYDATA:reason}"]}  
}  
if "\_grokparsefailure" in [tags] {  
drop {}  
}  
mutate {  
remove\_field =\> ["message"]  
}  
}

I have to deal with the last problem. I want to compare the field dst-ip with a ip.txt file, which contain one IP per line.  
I just moved this file to ip.csv.  
Here is the lines I add to the conf file, but it doesn't work.

```
translate {
	field => ["dst-ip"]
	destination => ["malicious"]
	dictionary_path => '/home/t0/full_ip.csv'
	refresh_interval => '1000'
}

```

Any help plz ? Thank you.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 15, 2019, 6:51pm UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/2 "2019-03-15T18:51:53Z")

</div>

> [@theo1991](#):
>
> but it doesn't work.

What don't you like about the result?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [March 15, 2019, 7:00pm UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/3 "2019-03-15T19:00:35Z")

</div>

Your grok patterns look [quite inefficient](https://www.elastic.co/blog/do-you-grok-grok) as they contain a lot of DATA and GREEDYDATA fields. It looks to me like your logs have a number of common fields and then end in a key-value list. It might be cleaner and more efficient to parse out the initial fields using a single grok pattern and in this store the full key-value list in a single field which you can then run a kv filter against.

It would help if you showed us what the data and the lookup file looks like as well as what the result is and what in this that is not like you expect.

---

<div class="post-metadata">

**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\
**Post date:** [March 15, 2019, 7:21pm UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/4 "2019-03-15T19:21:27Z")

</div>

There's no result in kibana, the field malicious doesn't appear.

I would like a new field in kibana called malicious, which is true when the dst-ip field and the ip in the full\_ip.csv match, and wrong when not.

---

<div class="post-metadata">

**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\
**Post date:** [March 15, 2019, 7:26pm UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/5 "2019-03-15T19:26:53Z")

</div>

I agree, I have a lot of GREEDYDATA, but I will adjust the filters later when the config file works.

I would like a new field in kibana called malicious, which is true when the dst-ip field and the ip in the full\_ip.csv match, and wrong when not.

Here is an example of my full\_ip.csv :  
8.8.8.8  
1.1.1.1  
2.2.2.2  
...

I think the problem doesn't come from my grok filter because it works well and kibana matches all the fields and are at the correct place.  
Maybe I have to add some ; in my full\_ip.csv ?  
Like this ?

8.8.8.8;TRUE  
1.1.1.1;TRUE  
etc ?

Thanks for your advice

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [March 15, 2019, 7:42pm UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/6 "2019-03-15T19:42:06Z")

</div>

If you only have the ip address in the lookup then you will get

```
 "malicious" => nil,

```

in the event. It's a csv, so if you want to set it to something else use

```
8.8.8.8,TRUE
```

---

<div class="post-metadata">

**Author:** ![theo1991](https://avatars.discourse-cdn.com/v4/letter/t/4da419/32.png) [@theo1991](https://discuss.elastic.co/u/theo1991)\
**Post date:** [March 15, 2019, 9:52pm UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/7 "2019-03-15T21:52:58Z")

</div>

Thank you ! It works

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 12, 2019, 9:53pm UTC](https://discuss.elastic.co/t/compare-fields-with-an-external-file/172492/8 "2019-04-12T21:53:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.