Grok stalling logstash

Elastic Stack Logstash
1

I am trying to parse via grok , a palo log in the format

<14>Apr 29 19:48:21 XXX-PANO-LOG2 1,2020/04/29 19:48:20,013101003497,TRAFFIC,end,2049,2020/04/29 19:48:19,123.456.78.910,X.X.X.X,0.0.0.0,0.0.0.0,Skype Stun Traffic to Outside - 20200227,,,stun,vsys1,dmz,outside,ethernet1/9,ethernet1/5,COMPANY_forwarding_profile,2020/04/29 19:48:19,37083153,1,3478,3478,0,0,0x50,udp,allow,19800,19800,0,330,2020/04/29 19:43:09,10,any,0,6736770790510724927,0x8000000000000000,United county,United States,0,330,0,aged-out,1148,0,0,0,,name-what-ng-fw1,from-policy,,,0,,0,,N/A,0,0,0,0,,0

Here is my filter ...

filter {
   #if ([message] =~ "User-Agent(.+)") {
   #   drop {}
   #}
       if [type] == "palo_syslog" {
      #FORMAT SYSLOG PRIORITY TO HUMAN READABLE FORMAT
      syslog_pri { }
    }
    grok {
        id => "grok_palo"
		#turn off grok timeout_millis
		timeout_millis => 0
     #break on match set to true
     break_on_match => true
     match => { "message" => "%{SYSLOG5424PRI}%{CISCOTIMESTAMP}\s*%{HOSTNAME:host_name}\s%{NUMBER},20%{DATESTAMP:receive_time},%{INT:serial},%{WORD:type},%{WORD:subtype},%{INT},20%{DATESTAMP:time_generated},%{IP:src},%{IP:dst},%{IP:natsrc},%{IP:natdst},%{DATA:rule},,,%{DATA:app},%{DATA:virtual_system},%{DATA:from},%{DATA:to},%{DATA:inbound_if},%{DATA:outbound_if},%{DATA:logset},%{DATA},%{INT:sessionid},%{INT:repeatcnt},%{POSINT:src_port},%{POSINT:dst_port},%{NUMBER},%{NUMBER},%{BASE16NUM:flags},%{WORD:protocol},%{WORD:action},%{NUMBER:bytes_total},%{NUMBER:bytes_sent},%{NUMBER:bytes_received},%{NUMBER:packets_total},20%{DATESTAMP:session_start},%{NUMBER:elapsed_time_sec},%{WORD},%{NUMBER},%{NUMBER},%{BASE16NUM},%{DATA:src_loc},%{DATA:dst_loc},%{NUMBER},%{NUMBER:packets_sent},%{NUMBER:packets_received},%{DATA:session_end_reason},%{DATA},,%{DATA:device_name},%{DATA:action_source},,,%{GREEDYDATA}"}
     #match anything else
     match => { "message" => "%{SYSLOG5424PRI}%{CISCOTIMESTAMP}\s*%{GREEDYDATA:message}"}

       }
    
}

Logstash hums along fine all the way to %{SYSLOG5424PRI}%{CISCOTIMESTAMP}\s*%{HOSTNAME:host_name}\s%{NUMBER} .. anything after that it starts stalling. I've tried DATA, DATESTAMPS and it just stalls . It appears on other versions of logstash, I am using 7.3. Any help is much appreciated

2

That is going to be insanely expensive to parse if it does not match because of the vast amount of backtracking that will be required. If those fields cannot contain a comma then I suggest replacing each one with

(?<app>[^,]*),

etc.

3

Thanks for your response , Badger. Can you explain what you mean by “‘these fields cannot contain a coma”. Would appreciate some examples. Again, thank you!

4

Nevermind, I figured out what you meant. Regardless even though that works in the debugger, logstash didn't budge.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.