Multiple match in Grok filter logstash is not working

Elastic Stack Logstash
1

Hello ,

My Grok filter looks like below

grok {
match => ["message","%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}%{SPACE}%{GREEDYDATA:ACT_OM_Session}","%{WORD:SBLSRVR_NAME}%{SPACE}%{WORD:SBLSRVR_STATE}","%{WORD:ServerName}%{SPACE}%{WORD:Comp_Alias}%{SPACE}%{WORD:CompStatus}%{SPACE}%{WORD:CompStartMode}%{SPACE}%{NUMBER:RunningTasks:int}%{SPACE}%{NUMBER:MaxTasks:int}"]
}

Output looks as below

Siebel Enterprise Applications Siebel Server Manager, Version 16.19.0.0 [23057] LANG_INDEPENDENT 
Copyright (c) 2008,2016, Oracle. All rights reserved.

The Programs (which include both the software and documentation) contain
proprietary information; they are provided under a license agreement containing
restrictions on use and disclosure and are also protected by copyright, patent,
and other intellectual and industrial property laws. Reverse engineering,
disassembly, or decompilation of the Programs, except to the extent required to
obtain interoperability with other independently created software or as specified
by law, is prohibited.

Oracle, JD Edwards, PeopleSoft, and Siebel are registered trademarks of
Oracle Corporation and/or its affiliates. Other names may be trademarks
of their respective owners.

If you have received this software in error, please notify Oracle Corporation
immediately at 1.800.ORACLE1.

Type "help" for list of commands, "help <topic>" for detailed help

Connected to 13 server(s) out of a total of 13 server(s) in the enterprise

srvrmgr> list active sessions show SV_NAME,CC_ALIAS,CG_ALIAS,TK_DISP_RUNSTATE,OM_LOGIN

SV_NAME       CC_ALIAS   CG_ALIAS   TK_DISP_RUNSTATE  OM_LOGIN                  
------------  ---------  ---------  ----------------  ------------------------  
D1220001028A  ServerMgr  System     Running                                     
D1220001028A  ServerMgr  System     Running                                     
D1220001028A  ServerMgr  System     Running                                     
D1220001028A  SRProc     SystemAux  Running                                     
D1220001028A  SRProc     SystemAux  Running           Forwarding Task           
D1220001028A  SRBroker   System     Running           SRVR:D1220001032A         
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running           SRVR:D1220001033A         
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running           SRVR:D1220001031A         
D1220001028A  SRBroker   System     Running           SRVR:D1220001022A         
D1220001028A  SRBroker   System     Running           SRVR:D1220001005A         
8A  SRBroker   System     Running           SRVR:D1220001019A         
D1220001028A  SRBroker   System     Running           SRVR:D1220001026A         
D1220001028A  SRBroker   System     Running           SRVR:D1220001030A         
D1220001028A  SRBroker   System     Running           SRVR:D1220001017A         
D1220001028A  SRBroker   System     Running           SRVR:D1220001023A         
D1220001028A  SRBroker   System     Running           SRVR:D1220001027A         
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running           Store task                
D1220001028A  SRBroker   System     Running                                     
D1220001028A  SRBroker   System     Running           Task creation task        
D1220001028A  SRBroker   System     Running           Response task             
D1220001028A  SRBroker   System     Running           Information caching task  
D1220001027A  ServerMgr  System     Running                                     
D1220001027A  ServerMgr  System     Running                                     
D1220001027A  ServerMgr  System     Running                                     
D1220001027A  SRProc     SystemAux  Running                                     
D1220001027A  SRProc     SystemAux  Running           Forwarding Task           
D1220001027A  SRBroker   System     Running           SRVR:D1220001032A         
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running           SRVR:D1220001031A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001033A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001019A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001017A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001026A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001022A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001024A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001005A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001023A         
D1220001027A  SRBroker   System     Running           SRVR:D1220001030A         
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running           SRVR:D1220001028A         
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running                                     
D1220001027A  SRBroker   System     Running           Store task                
D1220001027A  SRBroker   System     Running           Task creation task        
D1220001027A  SRBroker   System     Running           Response task             
               
D1220001031A  SRBroker   System     Running           Task creation task        


srvrmgr> list comp show SV_NAME,CC_ALIAS,CP_DISP_RUN_STATE,CP_STARTMODE,CP_NUM_RUN_TASKS,CP_MAX_TASKS order by CP_DISP_RUN_STATE

SV_NAME       CC_ALIAS        CP_DISP_RUN_STATE  CP_STARTMODE  CP_NUM_RUN_TASKS  CP_MAX_TASKS  
------------  --------------  -----------------  ------------  ----------------  ------------  
D1220001027A  FSMSrvr         Online             Auto          0                 20            
D1220001027A  ServerMgr       Running            Auto          3                 20            
D1220001027A  SRBroker        Running            Auto          29                100           
D1220001027A  SRProc          Running            Auto          2                 20            
srvrmgr> list server show SBLSRVR_NAME,SBLSRVR_STATE

SBLSRVR_NAME  SBLSRVR_STATE  
------------  -------------  
D1220001005A  Running        
D1220001022A  Running        

srvrmgr>

If you observe my output , it has 3 different kinds of patterns and hence I am trying to give 3 different grok filters in logstash.

Earlier I was using the below filter which works with two filters and when i add 3rd filter , it does not work.

grok {
    match => {
      "message" => [
      #Most specific grok: 
       "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}%{SPACE}%{WORD:ACT_OM_Session}",


	"%{WORD:SBLSRVR_NAME}%{SPACE}%{WORD:SBLSRVR_STATE}",
	    				"%{WORD:ServerName}%{SPACE}%{WORD:Comp_Alias}%{SPACE}%{WORD:CompStatus}%{SPACE}%{WORD:CompStartMode}%{SPACE}%{NUMBER:RunningTasks:int}%{SPACE}%{NUMBER:MaxTasks:int}"



        ]
  }

Any suggestions please ?

2

How do you get this data into logstash?
Are you sure that your event is really looking like the data you are showing on the input?
Seeing rubydebug stout output would also help us to see more details as the Logstash event on the input cannot be bigger than 4096B, and here you have over 7kB.

3

you can define all your patterns inside /opt/logstash/patterns and refer it inside ur config file . Lets say create a file in /opt/logstash/ex and define ur pattern like this

RUNNINGSTATUS

RUNNINGSTATUS "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}%{SPACE}%{WORD:ACT_OM_Session}"

COMPSTATUS

COMPSTATUS"%{WORD:SBLSRVR_NAME}%{SPACE}%{WORD:SBLSRVR_STATE}",
"%{WORD:ServerName}%{SPACE}%{WORD:Comp_Alias}%{SPACE}%{WORD:CompStatus}%{SPACE}%{WORD:CompStartMode}%{SPACE}%{NUMBER:RunningTasks:int}%{SPACE}%{NUMBER:MaxTasks:int}"

and in ur config file u can mention it as

filter {
grok {
patterns_dir => [ "/opt/patterns/ex" ]
match => [ "message", "%{RUNNINGSTATUS}",
"message", "%{COMPSTATUS}" ]
}
}

4

Hi @pastechecker , Thanks for your reply. I am getting this data using exec command . Actually i am running Exec plugin to run the command.

5

Hello @vinu89 , trying this and will update you.

6

Hello @vinu89 ,

I created a file patterns.txt under logstash

patterns.txt

RUNNINGSTATUS "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}%{SPACE}%{WORD:ACT_OM_Session}"
SRVRSTATUS "%{WORD:SBLSRVR_NAME}%{SPACE}%{WORD:SBLSRVR_STATE}"
COMPSTATUS "%{WORD:ServerName}%{SPACE}%{WORD:Comp_Alias}%{SPACE}%{WORD:CompStatus}%{SPACE}%{WORD:CompStartMode}%{SPACE}%{NUMBER:RunningTasks:int}%{SPACE}%{NUMBER:MaxTasks:int}"

input {

beats {
port => 5044
tags => ["srvr_logs"]
}

exec {
command => "E:\ELK\logstash\scripts\DEV_Srvrmgr.bat"
interval => 300
#type => "string"
tags => ["srvrmgr"]
codec => multiline {
# Grok pattern names are valid! :slight_smile:
pattern => "^D122"
negate => true
what => "previous"

    }


}

}

filter {

mutate {
remove_field => [ "host" ]
}

if "srvr_logs" in [tags]
{
grok {
match => {"message" => "%{WORD:EventType}%{SPACE}%{WORD:EventSubType}%{SPACE}%{INT:Severity}%{SPACE}%{WORD:SARMID}%{NOTSPACE}%{SPACE}%{PROG:EventDate}%{SPACE}%{TIME:EventTime}%{SPACE}%{GREEDYDATA:LogMessage}"}
}
}
else {

ruby {
code => '
event.set("message", event.get("message").split("\n"))
'
}
split {
field => "message"
}
grok {
patterns_dir => [ "E:\ELK\logstash\patterns\patterns.txt" ]

match => [ "message", "%{RUNNINGSTATUS}","message", "%{SRVSTATUS}","message", "%{COMPSTATUS}"]}
}
}

output {

if "srvr_logs" in [tags] {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "srvrlog-%{+YYYY.MM.dd}"
}

}

else
{
elasticsearch {
hosts => ["http://localhost:9200"]
index => "srvrmgr-%{+YYYY.MM.dd}"
}
}

}

But receiving grok_parse_failure . Anything wrong here ? Please help.

7

in patterns file you have to give hash symbol before RUNNINGstatus for eg (#RUNNINGSTATUS)
RUNNINGSTATUS "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}%{SPACE}%{WORD:ACT_OM_Session}" dont give in braces... can u also send what is the error

8

Hello @vinu89 ,

tried with a single pattern first as below ,

patterns.txt ,

#RUNNINGSTATUS "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}"

PFB screen shot and it is failing to grok it ,

image

Any suggestions please ?

9

no not like this
#RUNNINGSTATUS
RUNNINGSTATUS "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}"

and give only running status inside ur grok u shouldnt give other two as u have not defined

10

Hello @vinu89 , Great.

it works with single pattern now .

Pattern.txt

#COMPSTATUS
COMPSTATUS %{WORD:ServerName}%{SPACE}%{WORD:Comp_Alias}%{SPACE}%{WORD:CompStatus}%{SPACE}%{WORD:CompStartMode}%{SPACE}%{NUMBER:RunningTasks:int}%{SPACE}%{NUMBER:MaxTasks:int}

grok pattern

grok {
patterns_dir => [ "E:\ELK\logstash\patterns\patterns.txt" ]
match => {"message" => "%{COMPSTATUS}"}
}

Trying with multi pattern but failing to parse it :frowning: , Can you please help me out with grok expression ?

11

sure... can u send me all the patterns

12

Hello @vinu89 ,

Trying with below patterns.txt file ,
>
> #COMPSTATUS
> COMPSTATUS %{WORD:ServerName}%{SPACE}%{WORD:Comp_Alias}%{SPACE}%{WORD:CompStatus}%{SPACE}%{WORD:CompStartMode}%{SPACE}%{NUMBER:RunningTasks:int}%{SPACE}%{NUMBER:MaxTasks:int}
> #RUNNINGSTATUS RUNNINGSTATUS "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}"
> #SRVSTATUS
> SRVSTATUS "%{WORD:SBLSRVR_NAME}%{SPACE}%{WORD:SBLSRVR_STATE}"

Grok filter as below

grok {
patterns_dir => [ "E:\ELK\logstash\patterns\patterns.txt" ]
match => {"message" => "%{COMPSTATUS}",
"%{RUNNINGSTATUS}",
"%{SRVSTATUS}"}
}

but getting this error in logstash when started
>
> Sending Logstash logs to E:/ELK/logstash/logs which is now configured via log4j2.properties
> [2020-01-22T10:49:14,423][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
> [2020-01-22T10:49:14,442][INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.8.4"}
> [2020-01-22T10:49:16,150][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 53, column 39 (byte 1121) after filter {\r\n\r\nmutate \t{\r\n remove_field => [ "host" ]\r\n\t\t}\r\n\r\n\r\nif "srvr_logs" in [tags] \r\n{\r\ngrok {\r\nmatch => {"message" => "%{WORD:EventType}%{SPACE}%{WORD:EventSubType}%{SPACE}%{INT:Severity}%{SPACE}%{WORD:SARMID}%{NOTSPACE}%{SPACE}%{PROG:EventDate}%{SPACE}%{TIME:EventTime}%{SPACE}%{GREEDYDATA:LogMessage}"}\r\n\t}\r\n}\r\nelse {\r\n\r\nruby \t{\r\n code => '\r\n event.set("message", event.get("message").split("\n"))\r\n '\r\n\t\t\t\t}\r\n split \t{\r\n field => "message"\r\n\t\t\t\t}\r\ngrok {\r\npatterns_dir => [ "E:\ELK\logstash\patterns\patterns.txt" ]\r\nmatch => {"message" => "%{COMPSTATUS}"", :backtrace=>["E:/ELK/logstash/logstash-core/lib/logstash/compiler.rb:41:in compile_imperative'", "E:/ELK/logstash/logstash-core/lib/logstash/compiler.rb:49:in compile_graph'", "E:/ELK/logstash/logstash-core/lib/logstash/compiler.rb:11:in block in compile_sources'", "org/jruby/RubyArray.java:2577:in map'", "E:/ELK/logstash/logstash-core/lib/logstash/compiler.rb:10:in compile_sources'", "org/logstash/execution/AbstractPipelineExt.java:151:in initialize'", "E:/ELK/logstash/logstash-core/lib/logstash/pipeline.rb:22:in initialize'", "E:/ELK/logstash/logstash-core/lib/logstash/pipeline.rb:90:in initialize'", "E:/ELK/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:43:in block in execute'", "E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:96:in block in exclusive'", "org/jruby/ext/thread/Mutex.java:165:in synchronize'", "E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:96:in exclusive'", "E:/ELK/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:39:in execute'", "E:/ELK/logstash/logstash-core/lib/logstash/agent.rb:334:in block in converge_state'"]}
> [2020-01-22T10:49:16,603][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
> [2020-01-22T10:49:21,561][INFO ][logstash.runner ] Logstash shut down.

Please help ?

13

#COMPSTATUS
COMPSTATUS %{WORD:ServerName}%{SPACE}%{WORD:Comp_Alias}%{SPACE}%{WORD:CompStatus}%{SPACE}%{WORD:CompStartMode}%{SPACE}%{NUMBER:RunningTasks:int}%{SPACE}%{NUMBER:MaxTasks:int}
#RUNNINGSTATUS
RUNNINGSTATUS "%{WORD:ACT_SV_NAME}%{SPACE}%{WORD:ACT_CC_Alias}%{SPACE}%{WORD:ACT_CG_ALIAS}%{SPACE}%{WORD:ACT_Running_Status}"
#SRVSTATUS
SRVSTATUS "%{WORD:SBLSRVR_NAME}%{SPACE}%{WORD:SBLSRVR_STATE}"

filter {
grok {
patterns_dir => [ "/opt/patterns/ex" ]
match => [ "message", "%{RUNNINGSTATUS}",
"message", "%{COMPSTATUS}",
"message", "%{SRVSTATUS}" ]
}
}

14

Hello @vinu89 , Awesome .

it works .

Only a small correction is , we need not to place double quotes in the patters.txt file :slight_smile:

Have a Great day ahead.

Thanks alot !!

15

oh yea didnt note that in RUNNINGSTATUS SRVSTATUS...

ok....

16

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.