[SOLVED] Grok and Greedydata regex

[axelfelix (alexandre)]

Hi all,

I have an issue with my grok patterns. Some lines in my grok patterns contains GREEDYDATA regex. So they match before other lines and I cannot match logs with the fields that I want.

Could you tell me please a way to match what I want even with GREEDYDATA regex? Do you know if there is an order for the regex to be apply or could we put a kind of order?

Thanks in advance,
Alex

[anhlqn (Anh)]

Could you post some log samples and your current grok pattern?

[magnusbaeck (Magnus Bäck)]

Do you know if there is an order for the regex to be apply or could we put a kind of order?

If you list multiple grok expressions in the same grok filter they will be applied in the order given and the first match wins. Therefore you'll want to list the most specific expressions first.

[axelfelix (alexandre)]

Thanks but it is not solved, example below.

Patterns:

WLC_DATE (\w+ \d+ (\d+:){2}\d+.\d+)
NOT_DOUBLEDOT ([^:]*)
MNEMONIC_MESSAGE (\S+)
STATE (\w+\s+\w+\s+\w+)
ARP_FULL (ARP table is full.Could not add the new entry)


COMMON_WLC %{TIMESTAMP_ISO8601:syslog_host_time} %{IP:wlc_ip} %{WORD:severity} \*%{NOT_DOUBLEDOT:action}: %{WLC_DATE:wlc_date}: \#%{NOT_DOUBLEDOT:mnemonic}: %{MNEMONIC_MESSAGE:mnemonic_mess} Association identifier \d+ for client %{COMMONMAC:client_mac} \w+ %{STATE:state} \w+ %{COMMONMAC:to_delete} \w+ \w+ %{COMMONMAC:AP_mac} \w+ %{INT:slot} \w+ %{INT:vap}
ARP_WLC %{TIMESTAMP_ISO8601:syslog_host_time} %{IP:wlc_ip} %{WORD:severity} \*%{NOT_DOUBLEDOT:action}: %{WLC_DATE:wlc_date}: \#%{NOT_DOUBLEDOT:mnemonic}: %{MNEMONIC_MESSAGE:mnemonic_mess} %{ARP_FULL:arp_issue}
GENERAL_WLC %{TIMESTAMP_ISO8601:syslog_host_time} %{IP:wlc_ip} %{WORD:severity} \*%{NOT_DOUBLEDOT:action}: %{WLC_DATE:wlc_date}: \#%{NOT_DOUBLEDOT:mnemonic}: %{MNEMONIC_MESSAGE:mnemonic_mess} %{GREEDYDATA:message_info}

Logstash filter:

filter {
    if [type] == "cisco-wlc" {
        grok {
            match => {"message" => [ "%{COMMON_WLC}", "%{ARP_WLC}", "%{GENERAL_WLC}" ] }
            add_tag => [ "cisco_wlc" ]
            remove_field => [ "to_delete" ]
        }
        mutate {
            add_field => { "@source_host" => "%{wlc_ip}" }
            remove_field => [ "message" ]
        }
        dns {
            nameserver => "192.168.1.1"
            reverse => [ "@source_host" ]
            action => "replace"
        }
    }
}

Log example:
2017-01-16T05:48:09+01:00 10.126.4.202 err *spamApTask6: Jan 16 05:48:05.723: #LWAPP-3-INVALID_AID2: spam_api.c:1477 Association identifier 10 for client f9:00:54:32:b1:90 is already in use by f9:00:54:32:b1:90 for AP 10:f5:8b:65:0c:42 slot 1 vap 1

I already tried in GrokDebugger, and it is working. But only GENERAL_WLC match in production and not COMMON_WLC. Do you have an idea why ?

(I mean %{GREEDYDATA:message_info} take the data, but nerver %{COMMONMAC:client_mac} \w+ %{STATE:state} \w+ %{COMMONMAC:to_delete} \w+ \w+ %{COMMONMAC:AP_mac} \w+ %{INT:slot} \w+ %{INT:vap} fields)

The behavior is like GENERAL_WLC is matching before COMMON_WLC, it is possible ?

Logstash version: 2.3.0

Thanks in advance,
Alex

[magnusbaeck (Magnus Bäck)]

Does it work if you comment out the references to ARP_WLC and GENERAL_WLC in your grok filter?

[axelfelix (alexandre)]

I did what you ask and re-enable the message field to get it all.
I have _grokparsefailure all the time now, but it is pretty strange because my pattern is ok in grokdebugger website.

I didn't find the issue now...

[magnusbaeck (Magnus Bäck)]

Reduce your expression to the simplest possible. Make sure that works. Then start adding back pieces until it breaks again. That'll narrow down the problem.

[axelfelix (alexandre)]

Hey I feel abit dumb, I had to do that before you ask me to do...

Anyway, I found the issue and I'm really surprised, the problem is COMMONMAC pattern.

When I try in grokdebuger, it works. When I try in regex101, it works... But with logstash 2.3.0

So I will change the regex. But I take the answer, if there is an explanation regarding the COMMONMAC pattern.

Thanks,
Alex

[axelfelix (alexandre)]

Ok I found the problem, it was my bad.

I was using a pattern name with the same name for another pattern in another file...

No comment ...

Thanks all,
Alex