Problem with if statement in GROK filter

[vp096 (Valentin)]

Hello,

I send syslog data from a WiFi controller to ElasticSearch. This controller sends me different types of packets and I want to keep only the packets of this type:

Aug 25 11:19:50 2021 MC72-WI-AS1-06-B authmgr[3647]: <124006> <3647> <WARN> <MC72-WI-AS1-06-B 172.20.10.11>  {10069} TCP srcip=172.20.40.39 srcport=65178 dstip=87.117.121.3 dstport=443, action=permit, role=guest, policy=_syslog

So I made a filter as follows:

#Filtering data using grok parser
filter
{
        if [type] == "wifilogs" 
        {
            if "policy=_syslog" in [message]
            {
                grok 
                {    
                    match   =>   {  "message" => "(?<timestamp>%{MONTH} +%{MONTHDAY} %{TIME} %{YEAR}) %{HOSTNAME:ControllerName} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} (<%{NOTSPACE} %{IP:IPController}>|%{NOTSPACE}) %{NOTSPACE}  %{NOTSPACE} %{NOTSPACE:Protocol} srcip=%{IP:srcIP} srcport=%{NOTSPACE:srcPort} dstip=%{NOTSPACE:destIP} dstport=%{NOTSPACE:dstPort}, action=%{NOTSPACE:action}, role=%{NOTSPACE:role}, policy=%{NOTSPACE:ArubaPolicy}"}
                }
            }
        }
}

The grok filter works well ! No problem with that.
But I find that this filter does not work, because it still sends all packets to logstash. How can I keep only the packets that contain "policy=_syslog"?

Thanks in advance for your help.

[Cad]

Hi,

One possibilty is to product a grok pattern who recognised only syslog message. So for all other, it gonna product the tag _grokparsefailure so you can use this in a if expression in your output.

filter
{
    if [type] == "wifilogs" 
    {
#I edit the grok pattern, look a the end of it
        grok 
        {    
            match   =>   {  "message" => "(?<timestamp>%{MONTH} +%{MONTHDAY} %{TIME} %{YEAR}) %{HOSTNAME:ControllerName} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} (<%{NOTSPACE} %{IP:IPController}>|%{NOTSPACE}) %{NOTSPACE}  %{NOTSPACE} %{NOTSPACE:Protocol} srcip=%{IP:srcIP} srcport=%{NOTSPACE:srcPort} dstip=%{NOTSPACE:destIP} dstport=%{NOTSPACE:dstPort}, action=%{NOTSPACE:action}, role=%{NOTSPACE:role}, policy=_syslog"}
        }  
   }
}

output {
  if "_grokparsefailure" not in [tags] {
    elasticsearch {...}
  }
}

[aaron-nimocks]

Think this is what you are trying to do. If policy = syslog then grok. If not then drop the message.

#Filtering data using grok parser
filter
{
        if [type] == "wifilogs" 
        {
            if "policy=_syslog" in [message]
            {
                grok 
                {    
                    match   =>   {  "message" => "(?<timestamp>%{MONTH} +%{MONTHDAY} %{TIME} %{YEAR}) %{HOSTNAME:ControllerName} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} %{NOTSPACE} (<%{NOTSPACE} %{IP:IPController}>|%{NOTSPACE}) %{NOTSPACE}  %{NOTSPACE} %{NOTSPACE:Protocol} srcip=%{IP:srcIP} srcport=%{NOTSPACE:srcPort} dstip=%{NOTSPACE:destIP} dstport=%{NOTSPACE:dstPort}, action=%{NOTSPACE:action}, role=%{NOTSPACE:role}, policy=%{NOTSPACE:ArubaPolicy}"}
                }
            } else {
               drop { }
            }
        }
}

[vp096 (Valentin)]

Thanks for your help @Cad and @aaron-nimocks.

It's now working for me.

It works for me.

I would like to take this opportunity for a second question. I have to eat this kind of logs with grok :

2021-08-25 11:06:07,366 172.30.20.244 Syslog 1 1 0 Radius.Username=grs7yq5j,Radius.Framed-IP-Address=172.20.40.92,Radius.Start-Time=2021-08-25 11:03:43+02,Radius.End-Time=2021-08-25 11:03:51+02,Radius.Duration=8

Which GROK synthaxis should I use to catch the structured timestamp in this way?

I try something like this, but it doesn't work :

(?<timestamp>%{YEAR} -%{MONTHDAY} -%{DAY} %{TIME} )

Thanks in advance for your help!

[vp096 (Valentin)]

Il finally find the solution :

%{TIMESTAMP_ISO8601:timestamp}