Use field value to match custom Grok pattern

Elastic Stack Logstash
1

Hi,

I'm trying to parse some log (from cisco ironport) and I want to use a field value to access my custom patterns. To be clear there is my actual filter :

dissect {
mapping => {"message" => "<%{syslog5424pri}>%{mois} %{jour} %{heure}:%{minute}:%{seconde} %{log_type}: %{log_level}: %{iron_msg_type} %{iron_msg}"}
tag_on_failure => ["dpf_global"]
}
grok{
patterns_dir => ["/my/patterns/dir"]
break_on_match => true
match => {"iron_msg"=>"%{iron_msg_type}"}
tag_on_failure => ["_gpf_global"]
}

But Logstash return me he doesn't know about %{iron_msg_type}. So is it really possible to do this ? We it's possible, please help me.

2

Could you give us some raw input lines please? 3 or 4 would be good! Also can you post your patterns file?

3

Data sample

<142>Oct 05 05:08:33 Mail_Logs-syslog: Info: Start MID 100000000 ICID 200000000
<142>Oct 05 05:08:47 Mail_Logs-syslog: Info: Outbreak Rule: OUTBREAK_10000000 has threat level 5
<142>Oct 05 05:08:38 Mail_Logs-syslog: Info: Delivery start DCID 20000000 MID 100000000 to RID [0]

Custom Pattern:

Start (MID %{DATA:mid} ICID %{GREEDYDATA:icid})
Outbreak (%{GREEDYDATA}_%{DATA:outbreak_id} %{GREEDYDATA} %{DATA:threat_level})
Delivery (start DCID %{DATA:dcid} MID %{DATA:mid} to RID [%{GREEDYDATA:rid}])

4

Hi,

I see you have not defined a %{iron_msg_type} field in your pattern?

This is probably why the error is occuring.

5

No I don't have to define it because I want to use the value of this field (iron_msg_type) created in the dissect part to access my custom patterns.

6

No, your syntax is wrong as far as I can see. Your telling it to match "iron_msg" field to the pattern %{iron_msg_type} - you do not have an iron message pattern hence the error.

Try this instead:

Patterns file:
    START (MID %{DATA:mid} ICID %{GREEDYDATA:icid})
    OUTBREAK (%{GREEDYDATA}_%{DATA:outbreak_id} %{GREEDYDATA} %{DATA:threat_level})
    DELIVERY (start DCID %{DATA:dcid} MID %{DATA:mid} to RID [%{GREEDYDATA:rid}])


grok{
patterns_dir => ["/my/patterns/dir"]
break_on_match => true
match => {"iron_msg" => ["%{START}", "%{OUTBREAK}", "%{DELIVERY}"]}
tag_on_failure => ["_gpf_global"]
}

I've tested this and it works, but your patterns could do with a bit of fine tuning to make it nicer.

7

So it's impossible to use a field value to access the exact grok pattern directly?
I was thinking it could work like that:

Ex with a "Start" log
Dissect_process => iron_msg_type = "Start", iron_msg = "blablabla"
Then Grok Process => match => {"iron_msg"=>"Start"(the iron_msg_type value)}
Then go to my Start custom pattern.

8

Sorry I am not quite sure what you mean or what you are hoping to achieve? Maybe if you could explain what fields you want to end up with it would be easier?

9

Simply just don't want to specify all my custom patterns names in my grok matching like you did:

match => {"iron_msg" => ["%{START}", "%{OUTBREAK}", "%{DELIVERY}"]}

But do it like that instead:

match => {"iron_msg"=>"the value of the iron_msg_type field"}

So when [iron_msg_type] == "Start" Grok understand :

match => {"iron_msg"=>"Start"}

I'm not sure that is clear... This is my iron_msg_type field that define the custom pattern to use.

10

I understand now.

To be honest, I don't know, I've never tried to do that. I think it is evaluating it as a pattern rather than the taking it as a literal string.

I assume your log file varies so much that you would have to create many different patterns?

11

Exactly ! I think It would be more efficient in my way but I think you're right about this so I close this topic. Thanks a lot ! Have a nice day.

1 Like
12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.