Write a RegEx to match the event pattern in log file

[hamzeha]

Hi Everyone,
I have application log file which contains the application requests and responses, the complete request and response looks like the below, I tried different patterns using RegEx but unfortunately without any luck, can some one suggest what should I change :

pattern => "(?m)\b\d{4}-\d{2}-\d{2} \d{2}:\d{2}:\d{2},\d{3}\b.?\bRequest given by the user\b.?\brspDesc is\b(?!.\n)..?(?=\R|$)"

2023-05-11 00:20:26,103 [http-apr-7777-exec-46] INFO com.welcome.ws.AccountWebServiceImpl. - Request given by the user

<?xml version="1.0" encoding="UTF-8" standalone="yes"?>

<ns2:GetCustomerBalanceReq xmlns="url" xmlns:ns2="url">

testt
**
xls

ns2:ID/EID/5000000004/123456****1234</ns2:ID>
</ns2:GetCustomerBalanceReq>

2023-05-11 00:20:26,144 [http-apr-7777-exec-46] INFO com.welcome.svc.AccountService. - rspCode is:1001 and rspDesc is:Account or Media does not Exist

[Rios]

You can make something like this.

Save your lines in the sample.txt file.

input {
  file {
   path => "/path/sample.txt"
   start_position => beginning
      codec => multiline { 
      pattern => "%{TIMESTAMP_ISO8601}"
      negate => true
      what => "previous"
      }
   sincedb_path => "NUL"
   mode => "tail"
   
  }
}
filter {
    grok {
      match => { "message" => "^%{TIMESTAMP_ISO8601:timestamp}\s+\[%{DATA:thread}\]\s+%{LOGLEVEL:level}\s+%{DATA:method}\.\s+-\s+%{DATA:info}\s*(<%{GREEDYDATA:msg})?$" 
      }
	 
    }
  if [msg] =~ /^\?xml/ {
    ruby {
      code => "  event.set('[msg]', '<'+event.get('[msg]')  ) "
    }
  }
	
      date {
        match => ["timestamp", "ISO8601"]
        remove_field => [ "timestamp" ]
      }
  # prune { blacklist_names => [  "@version", "location", host, "event", "log", "message", "tags" ] } 
   
}

output {
    stdout {
        codec => rubydebug{ metadata => false}
    }
}

Result:

{
           "msg" => "<?xml version=\"1.0\" encoding=\"UTF-8\" standalone=\"yes\"?>\r\n<ns2:GetCustomerBalanceReq xmlns=\"url\" xmlns:ns2=\"url\">\r\n\r\ntestt\r\n**\r\nxls\r\n\r\nns2:ID/EID/5000000004/123456****1234</ns2:ID>\r\n</ns2:GetCustomerBalanceReq>\r\n</xml>\r",
         "level" => "INFO",
        "method" => "com.welcome.ws.AccountWebServiceImpl",
        "thread" => "http-apr-7777-exec-46",
          "info" => "Request given by the user",
    "@timestamp" => 2023-05-10T22:20:26.103Z
}
{
         "level" => "INFO",
        "method" => "com.welcome.svc.AccountService",
        "thread" => "http-apr-7777-exec-46",
          "info" => "rspCode is:1001 and rspDesc is:Account or Media does not Exist",
    "@timestamp" => 2023-05-10T22:20:26.144Z
}