Grok Filtering - Two Groks One Feed

Elastic Stack Logstash
1

I am trying to figure out how to make the Grok filtering work properly, but it appears that I am not having such luck. I looked at other documents but I can't seem to figure it out so hopefully someone smarter than me can assist me.

I have a Syslog feed coming from Panorama. Some of the feed messages are different than others. I wanted to filter on two things. One of them is the threat feeds and the other is from the honeypot feeds. Both are coming from the same source but I can't seem to filter it properly. The one for doing the threat feed works if used by itself. Thoughts? The pictures are the two file examples that I am trying to parse.

THREAT
THREAT.PNG449×1048 15.3 KB
HONEYPOT
HONEYPOT.PNG887×650 13.1 KB

This is my current Grok filter.
input {
syslog {
port => 5900
add_field => {
"source" => "panorama"
"sensor" => "panorama"
"sensor_type" => "internet"
}
}
}
filter {
if [source] == "panorama" {
grok {
match => { "message" => "%{DATA:future_use_1},%{DATESTAMP:receive_time},%{NUMBER:serial_number},%{SPACE}(?.*),%{WORD:subtype},%{DATA:future_use_2},%{DATESTAMP:generated_time},%{IP:source_ip},%{IP:destination_ip},%{IP:nat_source_ip},%{IP:nat_destination_ip},%{DATA:rule_name},%{DATA:source_user},%{DATA:destination_user},%{DATA:application},%{DATA:virtual_system},%{DATA:source_zone},%{DATA:destination_zone},%{DATA:ingress_interface},%{DATA:egress_interface},%{DATA:log_forwarding_profile},%{DATA:future_use_3},%{NUMBER:session_id},%{INT:repeat_count},%{INT:source_port},%{INT:destination_port},%{INT:nat_source_port},%{INT:nat_destination_port},%{BASE16NUM:flags},%{DATA:protocol},%{DATA:action},%{QUOTEDSTRING:miscellaneous},%{DATA:threat_id},%{DATA:category},%{WORD:severity},%{DATA:direction},%{INT:sequence_number},%{BASE16NUM:action_flags},%{DATA:source_location},%{DATA:destination_location},%{DATA:future_use_4},%{DATA:content_type},%{INT:pcap_id},%{DATA:filedigest},%{DATA:cloud},%{DATA:future_use_5},"}
add_field => {
"type" => "threatalert"
}
}

grok {
match => { "message" => "%{DATA:future_use_1},%{DATA:average_severity},%{DATA:severity},%{NUMBER:normalized_id},%{DATA:normalized_rule},%{DATA:signature_name},%{DATA:signature_id},%{DATA:rule_message},%{DATA:rule_id},%{DATA:event_subtype},%{NUMBER:event_id},%{NUMBER:event_count},%{DATA:source_user},%{IP:source_ip},%{INT:source_port},%{DATA:source_zone},%{DATA:geolocation_source},%{DATA:destination_user},%{IP:destination_ip},%{INT:destination_port},%{SPACE}(?.*),%{DATA:geolocation_destination},%{DATA:session_id},%{DATA:application},%{DATA:file_detail},%{DATA:category},%{DATA:interface},%{DATA:url},%{DATESTAMP:last_time},%{DATA:future_use_1},%{DATA:future_use_2},"}
add_field => {
"type" => "honeypot"
}
}

2

Try using another than "type" for your field. "type" is used in elasticsearch too and you can't have two types in one index since version 6.x

3

Thanks for the reply. I just saw on another post that someone had that and it worked for them so I guess I am confused. Do you have any suggestions or something I could try as an example?

4

Try naming the field "logtype" for example instead of "type". Like so:

add_field => {
"logtype" => "threatalert"
}

and

add_field => {
"logtype" => "honeypot"
}
5

Okay great. I made the change. I will see how it works. :slight_smile:

6

Well...that didn't work so my Grok filter must be wrong then.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.