Logstash multiline pattern not working

[Saravana37 (Saravana Selvaraj)]

Hello All ,

My logstash.conf file ..

input {
exec {
command => "E:\ELK\logstash\scripts\srvrmgr.bat"
interval => 300
codec => multiline {
# Grok pattern names are valid!
pattern => "^(D122|T122)"
negate => true
what => "previous"

                       }


 	        }
      }
      
filter  {

      if "ENTDEV" not in [message] and "ENTTEST" not in [message]
          {
            grok {
                   patterns_dir => [ "E:\ELK\logstash\patterns\patterns.txt" ]
                   match => ["message", "%{COMPSTATUS}",
                                        "message", "%{OMSTATUS}",
                                        "message", "%{PROCINFO}",
                                        "message", "%{SRVSTATUS}"]
                 }

                 if [ComponentStatus] 
                       {
                        mutate {
                        add_field => { 
                                "tags" => "COMPSTATUS"
                                       }
                                }
                        }

                            else if [PID] 
                            {
                               if [TaskId]
                                {
                                  mutate {
                                  add_field => { 
                                  "tags" => "OMSESSIONS"
                                               }
                                         }
                                 }
                               else  
                                 {
                                    mutate {
                                    add_field => { 
                                            "tags" => "PROCINFO"
                                                 }
                                            }
                                 }
                            }
                            else if [SIEBEL_SRV_STATUS] 
                            {
                            mutate {
                            add_field => { 
                                    "tags" => "SRVSTATUS"
                                           }
                                   }
                            }
                            }
							}
							

output 
{

if "D122" in [message]
        {
elasticsearch 	{
    hosts => ["https:XXXXXXXXXXX:8200"]
		    ssl => true
			ssl_certificate_verification => false
			cacert => "E:\ELK\ODForESearch\config\chain.pem"
			index => "devsrvrmgr-%{+YYYY.MM.dd}"      
user => "${es_usr}"
      password => "${es_pwd}"


				}
		}
        
else if "T122" in [message]
	{	
elasticsearch 		{
    hosts => ["https:XXXXXXXXX:8200"]
		    ssl => true
			ssl_certificate_verification => false
			cacert => "E:\ELK\ODForESearch\config\chain.pem"
			index => "testsrvrmgr-%{+YYYY.MM.dd}"
       user => "${es_usr}"
      password => "${es_pwd}"


  					}
				}
                
     }

and the output for the batch file looks as below ...

srvrmgr> list server show SBLSRVR_NAME,SBLSRVR_STATE

SBLSRVR_NAME  SBLSRVR_STATE  
------------  -------------  
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running        
D122000XXXXA  Running       
D122000XXXXA  Shutdown       

13 rows returned.

srvrmgr> 

Issue is i am not getting the last line "D122000XXXXA Shutdown " in Kibana because of this multiline pattern pattern => "^(D122|T122)" , Because as per this pattern the last line will be taken as below.
D122000XXXXA Shutdown

13 rows returned.

srvrmgr> 

FYI , GROK filter for this is
%{WORD:ServerName}%{SPACE}%{WORD:SIEBEL_SRV_STATUS}

So how can i specify the multiline pattern in Logstash so that it takes the last line without next new lines ? Please HELP.

[Badger]

@Saravana37

codec => multiline {
    # Grok pattern names are valid! :slight_smile:
    pattern => "^(D122|T122)"
    negate => true
    what => "previous"

That says that if a line does not start with either D122 or T122 then it should be combined with the previous line that does start with that pattern. What you are seeing is exactly what you asked for.

By default an exec output will combine all of stdout into a single [message] field. You could try removing the codec and then using

# Convert [message] into an array of lines
mutate { split { "message" => "
" } }

# Create one event per line
split { field => "message" }

if [message] !~ /^(D122|T122)/ { drop {} }

[Saravana37 (Saravana Selvaraj)]

Thank you. It is working.