Use field value to match custom Grok pattern


(PLUCHART Alexis) #1

Hi,

I'm trying to parse some log (from cisco ironport) and I want to use a field value to access my custom patterns. To be clear there is my actual filter :

dissect {
mapping => {"message" => "<%{syslog5424pri}>%{mois} %{jour} %{heure}:%{minute}:%{seconde} %{log_type}: %{log_level}: %{iron_msg_type} %{iron_msg}"}
tag_on_failure => ["dpf_global"]
}
grok{
patterns_dir => ["/my/patterns/dir"]
break_on_match => true
match => {"iron_msg"=>"%{iron_msg_type}"}
tag_on_failure => ["_gpf_global"]
}

But Logstash return me he doesn't know about %{iron_msg_type}. So is it really possible to do this ? We it's possible, please help me.


(Lewis Barclay) #2

Could you give us some raw input lines please? 3 or 4 would be good! Also can you post your patterns file?


(PLUCHART Alexis) #3

Data sample

<142>Oct 05 05:08:33 Mail_Logs-syslog: Info: Start MID 100000000 ICID 200000000
<142>Oct 05 05:08:47 Mail_Logs-syslog: Info: Outbreak Rule: OUTBREAK_10000000 has threat level 5
<142>Oct 05 05:08:38 Mail_Logs-syslog: Info: Delivery start DCID 20000000 MID 100000000 to RID [0]

Custom Pattern:

Start (MID %{DATA:mid} ICID %{GREEDYDATA:icid})
Outbreak (%{GREEDYDATA}_%{DATA:outbreak_id} %{GREEDYDATA} %{DATA:threat_level})
Delivery (start DCID %{DATA:dcid} MID %{DATA:mid} to RID [%{GREEDYDATA:rid}])


(Lewis Barclay) #4

Hi,

I see you have not defined a %{iron_msg_type} field in your pattern?

This is probably why the error is occuring.


(PLUCHART Alexis) #5

No I don't have to define it because I want to use the value of this field (iron_msg_type) created in the dissect part to access my custom patterns.


(Lewis Barclay) #6

No, your syntax is wrong as far as I can see. Your telling it to match "iron_msg" field to the pattern %{iron_msg_type} - you do not have an iron message pattern hence the error.

Try this instead:

Patterns file:
    START (MID %{DATA:mid} ICID %{GREEDYDATA:icid})
    OUTBREAK (%{GREEDYDATA}_%{DATA:outbreak_id} %{GREEDYDATA} %{DATA:threat_level})
    DELIVERY (start DCID %{DATA:dcid} MID %{DATA:mid} to RID [%{GREEDYDATA:rid}])


grok{
patterns_dir => ["/my/patterns/dir"]
break_on_match => true
match => {"iron_msg" => ["%{START}", "%{OUTBREAK}", "%{DELIVERY}"]}
tag_on_failure => ["_gpf_global"]
}

I've tested this and it works, but your patterns could do with a bit of fine tuning to make it nicer.


(PLUCHART Alexis) #7

So it's impossible to use a field value to access the exact grok pattern directly?
I was thinking it could work like that:

Ex with a "Start" log
Dissect_process => iron_msg_type = "Start", iron_msg = "blablabla"
Then Grok Process => match => {"iron_msg"=>"Start"(the iron_msg_type value)}
Then go to my Start custom pattern.


(Lewis Barclay) #8

Sorry I am not quite sure what you mean or what you are hoping to achieve? Maybe if you could explain what fields you want to end up with it would be easier?


(PLUCHART Alexis) #9

Simply just don't want to specify all my custom patterns names in my grok matching like you did:

match => {"iron_msg" => ["%{START}", "%{OUTBREAK}", "%{DELIVERY}"]}

But do it like that instead:

match => {"iron_msg"=>"the value of the iron_msg_type field"}

So when [iron_msg_type] == "Start" Grok understand :

match => {"iron_msg"=>"Start"}

I'm not sure that is clear... This is my iron_msg_type field that define the custom pattern to use.


(Lewis Barclay) #10

I understand now.

To be honest, I don't know, I've never tried to do that. I think it is evaluating it as a pattern rather than the taking it as a literal string.

I assume your log file varies so much that you would have to create many different patterns?


(PLUCHART Alexis) #11

Exactly ! I think It would be more efficient in my way but I think you're right about this so I close this topic. Thanks a lot ! Have a nice day.