Compare fields with an external file

Elastic Stack Logstash
1

Hi everybody,

I'm dealing with a problem since a few days.

First, I'm unable to delete the field message in kibana. I try with mutate, or directly in the grok filter, but there's always the field in kibana after a refresh.

Second point, I have a multiple grok filter, and the two first are closely equals, but there's a little difference : there's one with more fields (the first one have xlatedport xlatesport and xlatesrc) for example) and the second has not. In the current state, it only matches the first, the third and the 4th filter.

(To read it easily, I made some space after the comas. There's no space in my config file).

Thanks a lot.

filter {
grok {
match => { "message" => ["%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}<s_port>%{NUMBER:source-port}</s_port><d_port>%{NUMBER:destination-port}</d_port>%{WORD:action}%{NUMBER:xlatedport}%{NUMBER:xlatesport}%{IP:xlatesrc}%{WORD:service}<i_f_dir>%{WORD:iface-direction}</i_f_dir><i_f_name>%{WORD:iface-name}</i_f_name>%{NUMBER:rule-id}%{GREEDYDATA:product}<state%{GREEDYDATA:state}/>",

"%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}<s_port>%{NUMBER:source-port}</s_port><d_port>%{NUMBER:destination-port}</d_port>%{WORD:action}%{WORD:service}<i_f_dir>%{WORD:iface-direction}</i_f_dir><i_f_name>%{GREEDYDATA:iface-name}</i_f_name>%{NUMBER:rule-id}%{GREEDYDATA:product}<state%{GREEDYDATA:state}/>",

"1 %{GREEDYDATA:timestamp} %{GREEDYDATA:equipement} RT_FLOW - [%{GREEDYDATA:session} source-address="%{IP:origin-ip}" source-port="%{NUMBER:source-port}" destination-address="%{IP:dst-ip}" destination-port="%{NUMBER:destination-port}" service-name="%{GREEDYDATA:service-name}" nat-source-address="%{IP:nat-source-address}" nat-source-port="%{NUMBER:nat-source-port}" nat-destination-address="%{IP:nat-destination-address}" nat-destination-port="%{NUMBER:nat-destination-port}" src-nat-rule-type="%{GREEDYDATA:src-nat-rule-type}" src-nat-rule-name="%{GREEDYDATA:src-nat-rule-name}" dst-nat-rule-type="%{GREEDYDATA:dst-nat-rule-type}" dst-nat-rule-name="%{GREEDYDATA:dst-nat-rule-name}" protocol-id="%{NUMBER:protocol-id}" policy-name="%{GREEDYDATA:policy-name}" source-zone-name="%{WORD:source-zone-name}" destination-zone-name="%{WORD:destination-zone-name}" session-id-32="%{GREEDYDATA:session-id-32}" username="%{GREEDYDATA:username}" roles="%{GREEDYDATA:roles}" packet-incoming-interface="%{GREEDYDATA:packet-incoming-interface}" application="%{GREEDYDATA:application}" nested-application="%{GREEDYDATA:nested-application}" encrypted="%{GREEDYDATA:encrypted}"]",

"%{GREEDYDATA:firewall}: NetScreen device_id=%{GREEDYDATA:device} [Root]system-notification-00257(traffic): start_time="%{GREEDYDATA:timestamp}" duration=%{NUMBER:duration} policy_id=%{NUMBER:policy_id} service=%{WORD:service} proto=%{NUMBER:proto} src zone=%{WORD:src-zone} dst zone=%{WORD:dst-zone} action=%{WORD:action} sent=%{NUMBER:sent} rcvd=%{NUMBER:rcvd} src=%{IP:src-ip} dst=%{IP:dst-ip} src_port=%{NUMBER:source-port} dst_port=%{NUMBER:destination-port} session_id=%{NUMBER:session-id} reason=%{GREEDYDATA:reason}"]}
break_on_match => "false"
remove_field => ["message"]
}

}

Edit :slightly_smiling_face:

It seems I find the right solution.

filter {
grok {
match => { "message" => ["%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}<s_port>%{NUMBER:source-port}</s_port><d_port>%{NUMBER:destination-port}</d_port>%{WORD:action}%{NUMBER:xlatedport}%{NUMBER:xlatesport}%{IP:xlatesrc}%{WORD:service}<i_f_dir>%{WORD:iface-direction}</i_f_dir><i_f_name>%{WORD:iface-name}</i_f_name>%{NUMBER:rule-id}%{GREEDYDATA:product}<state%{GREEDYDATA:state}/>",

"%{NUMBER:timestamp}%{IP:device-ip}%{IP:src-ip}%{IP:dst-ip}<s_port>%{NUMBER:source-port}</s_port><d_port>%{NUMBER:destination-port}</d_port>%{WORD:action}%{WORD:service}<i_f_dir>%{WORD:iface-direction}</i_f_dir><i_f_name>%{GREEDYDATA:iface-name}</i_f_name>%{NUMBER:rule-id}%{GREEDYDATA:product}<state%{GREEDYDATA:state}/>",

"1 %{GREEDYDATA:timestamp} %{GREEDYDATA:equipement} RT_FLOW - [%{GREEDYDATA:session} source-address="%{IP:origin-ip}" source-port="%{NUMBER:source-port}" destination-address="%{IP:dst-ip}" destination-port="%{NUMBER:destination-port}" service-name="%{GREEDYDATA:service-name}" nat-source-address="%{IP:nat-source-address}" nat-source-port="%{NUMBER:nat-source-port}" nat-destination-address="%{IP:nat-destination-address}" nat-destination-port="%{NUMBER:nat-destination-port}" src-nat-rule-type="%{GREEDYDATA:src-nat-rule-type}" src-nat-rule-name="%{GREEDYDATA:src-nat-rule-name}" dst-nat-rule-type="%{GREEDYDATA:dst-nat-rule-type}" dst-nat-rule-name="%{GREEDYDATA:dst-nat-rule-name}" protocol-id="%{NUMBER:protocol-id}" policy-name="%{GREEDYDATA:policy-name}" source-zone-name="%{WORD:source-zone-name}" destination-zone-name="%{WORD:destination-zone-name}" session-id-32="%{GREEDYDATA:session-id-32}" username="%{GREEDYDATA:username}" roles="%{GREEDYDATA:roles}" packet-incoming-interface="%{GREEDYDATA:packet-incoming-interface}" application="%{GREEDYDATA:application}" nested-application="%{GREEDYDATA:nested-application}" encrypted="%{GREEDYDATA:encrypted}"]",

"%{GREEDYDATA:firewall}: NetScreen device_id=%{GREEDYDATA:device} [Root]system-notification-00257(traffic): start_time="%{GREEDYDATA:timestamp}" duration=%{NUMBER:duration} policy_id=%{NUMBER:policy_id} service=%{WORD:service} proto=%{NUMBER:proto} src zone=%{WORD:src-zone} dst zone=%{WORD:dst-zone} action=%{WORD:action} sent=%{NUMBER:sent} rcvd=%{NUMBER:rcvd} src=%{IP:src-ip} dst=%{IP:dst-ip} src_port=%{NUMBER:source-port} dst_port=%{NUMBER:destination-port} session_id=%{NUMBER:session-id} reason=%{GREEDYDATA:reason}"]}
}
if "_grokparsefailure" in [tags] {
drop {}
}
mutate {
remove_field => ["message"]
}
}

I have to deal with the last problem. I want to compare the field dst-ip with a ip.txt file, which contain one IP per line.
I just moved this file to ip.csv.
Here is the lines I add to the conf file, but it doesn't work.

translate {
	field => ["dst-ip"]
	destination => ["malicious"]
	dictionary_path => '/home/t0/full_ip.csv'
	refresh_interval => '1000'
}

Any help plz ? Thank you.

2

What don't you like about the result?

3

Your grok patterns look quite inefficient as they contain a lot of DATA and GREEDYDATA fields. It looks to me like your logs have a number of common fields and then end in a key-value list. It might be cleaner and more efficient to parse out the initial fields using a single grok pattern and in this store the full key-value list in a single field which you can then run a kv filter against.

It would help if you showed us what the data and the lookup file looks like as well as what the result is and what in this that is not like you expect.

4

There's no result in kibana, the field malicious doesn't appear.

I would like a new field in kibana called malicious, which is true when the dst-ip field and the ip in the full_ip.csv match, and wrong when not.

5

I agree, I have a lot of GREEDYDATA, but I will adjust the filters later when the config file works.

I would like a new field in kibana called malicious, which is true when the dst-ip field and the ip in the full_ip.csv match, and wrong when not.

Here is an example of my full_ip.csv :
8.8.8.8
1.1.1.1
2.2.2.2
...

I think the problem doesn't come from my grok filter because it works well and kibana matches all the fields and are at the correct place.
Maybe I have to add some ; in my full_ip.csv ?
Like this ?

8.8.8.8;TRUE
1.1.1.1;TRUE
etc ?

Thanks for your advice

6

If you only have the ip address in the lookup then you will get

 "malicious" => nil,

in the event. It's a csv, so if you want to set it to something else use

8.8.8.8,TRUE
7

Thank you ! It works

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.