Logstash Filter: I want to match this expression using "if" condition

Hello Dears,

My expression:

<?xml version="1.0" encoding="UTF-8"?>

I want to match this expr if "message" field contains it.

My Filter:

filter {
        
if "\\<\\?xml\\ version="1\\.0"\\encoding="UTF-8"?>" in [message]{
              do something......
}
else { drop{} }
}

Thanks

You are escaping things you do not need to and not escaping the things you do (the double quotes). Try

if '<?xml version="1.0" encoding="UTF-8"?>' in ...

@Badger Thanks for your response, I tried your solution but it is not working fine with me.

Expected output:

But I got!!
image

@Badger can you help me?

Can you post your full configuration? I tested the expression as posted and it works.

input { generator { lines => ['xxxx<?xml version="1.0" encoding="UTF-8"?>xxxx'] count => 1 codec => "line" } }
filter {
 if '<?xml version="1.0" encoding="UTF-8"?>' in [message] {
   mutate {
     add_tag => [ "matched" ]
   }
 } else { drop{} }
}    
output {
  stdout { codec =>  "json" }
}

Output

{
    "message": "xxxx<?xml version=\"1.0\" encoding=\"UTF-8\"?>xxxx",
    "host": "Aarons-MBP",
    "@version": "1",
    "@timestamp": "2021-12-21T12:56:51.172Z",
    "tags": [
        "matched"
    ]
}

Thanks @aaron-nimocks for your response,

My full configuration

input {
    file {
       path => "/etc/logstash/logsample.txt"
       start_position => "beginning"
    }
}


filter {
  if '<?xml version="1.0" encoding="UTF-8"?>' in [message] {
     xml {
        source => "message"
        store_xml => false
        remove_namespaces => true
        force_array => false       
        xpath => [
            "//Envelope/Header/Header/RequestID/text()", "RequestID",
            "//Body/PaymentValidationResponse/StatusDescription/text()", "StatusDescription",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/InvoiceCode/text()", "InvoiceCode",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/SupplierID/text()", "SupplierID",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/SupplierName/text()", "SupplierName",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/Amount/text()", "Amount",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/StatusCode/text()", "StatusCode",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/ErrorMessage/text()", "ErrorMessage"
        ]
     }
    } else { drop{} }

}

output {
        stdout { codec => rubydebug }       
}

I'm not the best with XML but I would verify your XML is valid. When I tested it based off an image to conversion tool of what you posted prior it wasn't passing validation. There maybe more to the XML than what was in your image though.

If you add parse_options => "strict" and you get a _xmlparsefailure you know it's not valid.

yes my XML is valid, I tested it before adding my condition.

If you can post your XML I can see if I can figure it out. But it appears what I extracted from your image isn't correct.

@aaron-nimocks XML Sample:

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Header>
      <Header xmlns="https://sih.sadad.com/PaymentValidation/defs" xmlns:ns2="https://sih.sadad.com/common">
         <ns2:RequestID>1a7f81ac-355f-11ec-bc47-3b3678cb106a</ns2:RequestID>
      </Header>
   </soap:Header>
   <soap:Body>
      <PaymentValidationResponse xmlns="https://sih.sadad.com/PaymentValidation/defs" xmlns:ns2="https://sih.sadad.com/common">
         <StatusCode>0</StatusCode>
         <StatusDescription>Success</StatusDescription>
         <PaymentValidations>
            <PaymentValidation>
               <InvoiceCode>175810975800</InvoiceCode>
               <SupplierID>649</SupplierID>
               <SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName>
               <Amount>2552.23</Amount>
               <StatusCode>0</StatusCode>
            </PaymentValidation>
         </PaymentValidations>
      </PaymentValidationResponse>
   </soap:Body>
</soap:Envelope>

I was able to get it to work with this config:

input {
  generator {
    message => '<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Header>
      <Header xmlns="https://sih.sadad.com/PaymentValidation/defs" xmlns:ns2="https://sih.sadad.com/common">
         <ns2:RequestID>1a7f81ac-355f-11ec-bc47-3b3678cb106a</ns2:RequestID>
      </Header>
   </soap:Header>
   <soap:Body>
      <PaymentValidationResponse xmlns="https://sih.sadad.com/PaymentValidation/defs" xmlns:ns2="https://sih.sadad.com/common">
         <StatusCode>0</StatusCode>
         <StatusDescription>Success</StatusDescription>
         <PaymentValidations>
            <PaymentValidation>
               <InvoiceCode>175810975800</InvoiceCode>
               <SupplierID>649</SupplierID>
               <SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName>
               <Amount>2552.23</Amount>
               <StatusCode>0</StatusCode>
            </PaymentValidation>
         </PaymentValidations>
      </PaymentValidationResponse>
   </soap:Body>
</soap:Envelope>'
  count => 1
  codec => multiline {
    pattern => '<?xml version="1.0" encoding="UTF-8"?>'
    what => previous
    negate => true
   }

  }

}

filter {

if '<?xml version="1.0" encoding="UTF-8"?>' in [message] {

  xml {
   source => "message"
   store_xml => true
   target => "doc"
   remove_namespaces => true
   force_array => false
   xpath => [
            "//Envelope/Header/Header/RequestID/text()", "RequestID",
            "//Body/PaymentValidationResponse/StatusDescription/text()", "StatusDescription",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/InvoiceCode/text()", "InvoiceCode",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/SupplierID/text()", "SupplierID",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/SupplierName/text()", "SupplierName",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/Amount/text()", "Amount",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/StatusCode/text()", "StatusCode",
            "//Body/PaymentValidationResponse/PaymentValidations/PaymentValidation/ErrorMessage/text()", "ErrorMessage"
        ]
 
  }
}

}

output {

stdout{}
}

That gave me this result:

[INFO ] 2021-12-21 10:07:07.532 [Api Webserver] agent - Successfully started Logstash API endpoint {:port=>9601}
/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/awesome_print-1.7.0/lib/awesome_print/formatters/base_formatter.rb:31: warning: constant ::Fixnum is deprecated
{
               "Amount" => "2552.23",
             "@version" => "1",
          "InvoiceCode" => "175810975800",
    "StatusDescription" => "Success",
                 "host" => "wtlsuv403.xxxx.com",
         "SupplierName" => "MEDITERRANEAN SHIPPING COMPANY SAUDI",
           "StatusCode" => "0",
                  "doc" => {
              "Body" => {
            "PaymentValidationResponse" => {
                         "xmlns:ns2" => "https://sih.sadad.com/common",
                 "StatusDescription" => "Success",
                        "StatusCode" => "0",
                "PaymentValidations" => {
                    "PaymentValidation" => {
                         "InvoiceCode" => "175810975800",
                        "SupplierName" => "MEDITERRANEAN SHIPPING COMPANY SAUDI",
                          "SupplierID" => "649",
                          "StatusCode" => "0",
                              "Amount" => "2552.23"
                    }
                },
                             "xmlns" => "https://sih.sadad.com/PaymentValidation/defs"
            }
        },
        "xmlns:soap" => "http://schemas.xmlsoap.org/soap/envelope/",
            "Header" => {
            "Header" => {
                "xmlns:ns2" => "https://sih.sadad.com/common",
                "RequestID" => "1a7f81ac-355f-11ec-bc47-3b3678cb106a",
                    "xmlns" => "https://sih.sadad.com/PaymentValidation/defs"
            }
        }
    },
              "message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\">\n   <soap:Header>\n      <Header xmlns=\"https://sih.sadad.com/PaymentValidation/defs\" xmlns:ns2=\"https://sih.sadad.com/common\">\n         <ns2:RequestID>1a7f81ac-355f-11ec-bc47-3b3678cb106a</ns2:RequestID>\n      </Header>\n   </soap:Header>\n   <soap:Body>\n      <PaymentValidationResponse xmlns=\"https://sih.sadad.com/PaymentValidation/defs\" xmlns:ns2=\"https://sih.sadad.com/common\">\n         <StatusCode>0</StatusCode>\n         <StatusDescription>Success</StatusDescription>\n         <PaymentValidations>\n            <PaymentValidation>\n               <InvoiceCode>175810975800</InvoiceCode>\n               <SupplierID>649</SupplierID>\n               <SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName>\n               <Amount>2552.23</Amount>\n               <StatusCode>0</StatusCode>\n            </PaymentValidation>\n         </PaymentValidations>\n      </PaymentValidationResponse>\n   </soap:Body>\n</soap:Envelope>",
                 "tags" => [
        [0] "multiline"
    ],
            "RequestID" => "1a7f81ac-355f-11ec-bc47-3b3678cb106a",
           "SupplierID" => "649",
           "@timestamp" => 2021-12-21T15:07:05.930Z
}
[INFO ] 2021-12-21 10:07:11.690 [LogStash::Runner] runner - Logstash shut down.

1 Like

If it is pretty-printed in the file then you will need a multiline codec to create a single event from the entire XML document.

@AquaX @Badger I have logs did not contain XML format in [message] field, so I want to select the logs that contains XML only.

You will still need to use a multiline codec to potentially grab the XML. If the multiline condition does not match (like in my example) then it will not combine the lines.

I added this part:

  codec => multiline {
    pattern => '<?xml version="1.0" encoding="UTF-8"?>'
    what => previous
    negate => true
   }

to my input plugin, but I still have the same issue.

Can you post a complete logstash log and example file?
I want to try and re-create your issue with your data.

@AquaX what do you mean "complete logstash log"?

example file:

<?xml version="1.0" encoding="UTF-8"?>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">
   <soap:Header>
      <Header xmlns="https://sih.sadad.com/PaymentValidation/defs" xmlns:ns2="https://sih.sadad.com/common">
         <ns2:RequestID>1a7f81ac-355f-11ec-bc47-3b3678cb106a</ns2:RequestID>
      </Header>
   </soap:Header>
   <soap:Body>
      <PaymentValidationResponse xmlns="https://sih.sadad.com/PaymentValidation/defs" xmlns:ns2="https://sih.sadad.com/common">
         <StatusCode>0</StatusCode>
         <StatusDescription>Success</StatusDescription>
         <PaymentValidations>
            <PaymentValidation>
               <InvoiceCode>175810975800</InvoiceCode>
               <SupplierID>649</SupplierID>
               <SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName>
               <Amount>2552.23</Amount>
               <StatusCode>0</StatusCode>
            </PaymentValidation>
         </PaymentValidations>
      </PaymentValidationResponse>
   </soap:Body>
</soap:Envelope>

<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>e603bb46-3636-11ec-bf4a-0afc4a1d0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>217461255537</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>451.0</Amount><StatusCode>90005</StatusCode><ErrorMessage>Partial payments are not permitted</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>e0db96fa-339f-11ec-a054-0afc4a1e0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>890452823685</InvoiceCode><SupplierID>657</SupplierID><SupplierName>Wataniya insurance</SupplierName><Amount>1.0</Amount><StatusCode>90006</StatusCode><ErrorMessage>Amount entered is less than the minimum (partial payment) payable</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>0260f0c6-363a-11ec-bf4a-0afc4a1d0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>178027290132</InvoiceCode><SupplierID>512</SupplierID><SupplierName>Saudi Ceramic Company</SupplierName><Amount>3340.28</Amount><StatusCode>90007</StatusCode><ErrorMessage>Amount entered is greater than the invoice outstanding amount</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>4fea79da-34ee-11ec-a402-4b524c5eabc0</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>492369966504</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>649.83</Amount><StatusCode>90008</StatusCode><ErrorMessage>Payment failure, please try again</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:RequestID>ff9e674f-355e-11ec-abff-6350b5a50346</ns2:RequestID></Header></soap:Header><soap:Body><BillQueryResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><StatusCode>0</StatusCode><StatusDescription>Success</StatusDescription><Paging><ns2:Page>1</ns2:Page><ns2:PageSize>1</ns2:PageSize><ns2:TotalNumRecords>1</ns2:TotalNumRecords></Paging><Bills><Bill><InvoiceCode>175810975800</InvoiceCode><InvoiceNumber>SADMMPM210026996</InvoiceNumber><SupplierID>649</SupplierID><SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName><BuyerName>ARABCO LOGISTICS (SALEH ABDUL AZIZ AL AMRY CARGO)</BuyerName><DueDate>2021-10-19T00:00:00</DueDate><DueAmount>2552.23</DueAmount><BillCategory>EXACT</BillCategory><BillStatus><LocalizedBillStatus><ns2:Locale>ar-SA</ns2:Locale><ns2:Value>معلّقة</ns2:Value></LocalizedBillStatus><LocalizedBillStatus><ns2:Locale>en-US</ns2:Locale><ns2:Value>Pending</ns2:Value></LocalizedBillStatus></BillStatus><MinPaymentAmount>0.00</MinPaymentAmount><MaxPaymentAmount>0.00</MaxPaymentAmount></Bill></Bills></BillQueryResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>The SADAD Bills ID provided does not exist</faultstring><detail><BillQueryFault xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:StatusCode>90000</ns2:StatusCode><ns2:ErrorCode>EM1</ns2:ErrorCode><ns2:ErrorMsg>The SADAD Bills ID provided does not exist</ns2:ErrorMsg></BillQueryFault></detail></soap:Fault></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>4fea79da-34ee-11ec-a402-4b524c5eabc0</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>492369966504</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>649.83</Amount><StatusCode>90008</StatusCode><ErrorMessage>Payment failure, please try again</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:RequestID>ff9e674f-355e-11ec-abff-6350b5a50346</ns2:RequestID></Header></soap:Header><soap:Body><BillQueryResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><StatusCode>0</StatusCode><StatusDescription>Success</StatusDescription><Paging><ns2:Page>1</ns2:Page><ns2:PageSize>1</ns2:PageSize><ns2:TotalNumRecords>1</ns2:TotalNumRecords></Paging><Bills><Bill><InvoiceCode>175810975800</InvoiceCode><InvoiceNumber>SADMMPM210026996</InvoiceNumber><SupplierID>649</SupplierID><SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName><BuyerName>ARABCO LOGISTICS (SALEH ABDUL AZIZ AL AMRY CARGO)</BuyerName><DueDate>2021-10-19T00:00:00</DueDate><DueAmount>2552.23</DueAmount><BillCategory>EXACT</BillCategory><BillStatus><LocalizedBillStatus><ns2:Locale>ar-SA</ns2:Locale><ns2:Value>معلّقة</ns2:Value></LocalizedBillStatus><LocalizedBillStatus><ns2:Locale>en-US</ns2:Locale><ns2:Value>Pending</ns2:Value></LocalizedBillStatus></BillStatus><MinPaymentAmount>0.00</MinPaymentAmount><MaxPaymentAmount>0.00</MaxPaymentAmount></Bill></Bills></BillQueryResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>The SADAD Bills ID provided does not exist</faultstring><detail><BillQueryFault xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:StatusCode>90000</ns2:StatusCode><ns2:ErrorCode>EM1</ns2:ErrorCode><ns2:ErrorMsg>The SADAD Bills ID provided does not exist</ns2:ErrorMsg></BillQueryFault></detail></soap:Fault></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>4fea79da-34ee-11ec-a402-4b524c5eabc0</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>492369966504</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>649.83</Amount><StatusCode>90008</StatusCode><ErrorMessage>Payment failure, please try again</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:RequestID>ff9e674f-355e-11ec-abff-6350b5a50346</ns2:RequestID></Header></soap:Header><soap:Body><BillQueryResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><StatusCode>0</StatusCode><StatusDescription>Success</StatusDescription><Paging><ns2:Page>1</ns2:Page><ns2:PageSize>1</ns2:PageSize><ns2:TotalNumRecords>1</ns2:TotalNumRecords></Paging><Bills><Bill><InvoiceCode>175810975800</InvoiceCode><InvoiceNumber>SADMMPM210026996</InvoiceNumber><SupplierID>649</SupplierID><SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName><BuyerName>ARABCO LOGISTICS (SALEH ABDUL AZIZ AL AMRY CARGO)</BuyerName><DueDate>2021-10-19T00:00:00</DueDate><DueAmount>2552.23</DueAmount><BillCategory>EXACT</BillCategory><BillStatus><LocalizedBillStatus><ns2:Locale>ar-SA</ns2:Locale><ns2:Value>معلّقة</ns2:Value></LocalizedBillStatus><LocalizedBillStatus><ns2:Locale>en-US</ns2:Locale><ns2:Value>Pending</ns2:Value></LocalizedBillStatus></BillStatus><MinPaymentAmount>0.00</MinPaymentAmount><MaxPaymentAmount>0.00</MaxPaymentAmount></Bill></Bills></BillQueryResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>The SADAD Bills ID provided does not exist</faultstring><detail><BillQueryFault xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:StatusCode>90000</ns2:StatusCode><ns2:ErrorCode>EM1</ns2:ErrorCode><ns2:ErrorMsg>The SADAD Bills ID provided does not exist</ns2:ErrorMsg></BillQueryFault></detail></soap:Fault></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>The SADAD Bills ID provided does not exist</faultstring><detail><BillQueryFault xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:StatusCode>90000</ns2:StatusCode><ns2:ErrorCode>EM1</ns2:ErrorCode><ns2:ErrorMsg>The SADAD Bills ID provided does not exist</ns2:ErrorMsg></BillQueryFault></detail></soap:Fault></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>4fea79da-34ee-11ec-a402-4b524c5eabc0</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>492369966504</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>649.83</Amount><StatusCode>90008</StatusCode><ErrorMessage>Payment failure, please try again</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:RequestID>ff9e674f-355e-11ec-abff-6350b5a50346</ns2:RequestID></Header></soap:Header><soap:Body><BillQueryResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><StatusCode>0</StatusCode><StatusDescription>Success</StatusDescription><Paging><ns2:Page>1</ns2:Page><ns2:PageSize>1</ns2:PageSize><ns2:TotalNumRecords>1</ns2:TotalNumRecords></Paging><Bills><Bill><InvoiceCode>175810975800</InvoiceCode><InvoiceNumber>SADMMPM210026996</InvoiceNumber><SupplierID>649</SupplierID><SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName><BuyerName>ARABCO LOGISTICS (SALEH ABDUL AZIZ AL AMRY CARGO)</BuyerName><DueDate>2021-10-19T00:00:00</DueDate><DueAmount>2552.23</DueAmount><BillCategory>EXACT</BillCategory><BillStatus><LocalizedBillStatus><ns2:Locale>ar-SA</ns2:Locale><ns2:Value>معلّقة</ns2:Value></LocalizedBillStatus><LocalizedBillStatus><ns2:Locale>en-US</ns2:Locale><ns2:Value>Pending</ns2:Value></LocalizedBillStatus></BillStatus><MinPaymentAmount>0.00</MinPaymentAmount><MaxPaymentAmount>0.00</MaxPaymentAmount></Bill></Bills></BillQueryResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>The SADAD Bills ID provided does not exist</faultstring><detail><BillQueryFault xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:StatusCode>90000</ns2:StatusCode><ns2:ErrorCode>EM1</ns2:ErrorCode><ns2:ErrorMsg>The SADAD Bills ID provided does not exist</ns2:ErrorMsg></BillQueryFault></detail></soap:Fault></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>4fea79da-34ee-11ec-a402-4b524c5eabc0</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Ahmed Moawad</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>492369966504</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>649.83</Amount><StatusCode>90008</StatusCode><ErrorMessage>Payment failure, please try again</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:RequestID>ff9e674f-355e-11ec-abff-6350b5a50346</ns2:RequestID></Header></soap:Header><soap:Body><BillQueryResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><StatusCode>0</StatusCode><StatusDescription>Success</StatusDescription><Paging><ns2:Page>1</ns2:Page><ns2:PageSize>1</ns2:PageSize><ns2:TotalNumRecords>1</ns2:TotalNumRecords></Paging><Bills><Bill><InvoiceCode>175810975800</InvoiceCode><InvoiceNumber>SADMMPM210026996</InvoiceNumber><SupplierID>649</SupplierID><SupplierName>MEDITERRANEAN SHIPPING COMPANY SAUDI</SupplierName><BuyerName>ARABCO LOGISTICS (SALEH ABDUL AZIZ AL AMRY CARGO)</BuyerName><DueDate>2021-10-19T00:00:00</DueDate><DueAmount>2552.23</DueAmount><BillCategory>EXACT</BillCategory><BillStatus><LocalizedBillStatus><ns2:Locale>ar-SA</ns2:Locale><ns2:Value>معلّقة</ns2:Value></LocalizedBillStatus><LocalizedBillStatus><ns2:Locale>en-US</ns2:Locale><ns2:Value>Pending</ns2:Value></LocalizedBillStatus></BillStatus><MinPaymentAmount>0.00</MinPaymentAmount><MaxPaymentAmount>0.00</MaxPaymentAmount></Bill></Bills></BillQueryResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Body><soap:Fault><faultcode>soap:Server</faultcode><faultstring>The SADAD Bills ID provided does not exist</faultstring><detail><BillQueryFault xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/BillQuery/defs"><ns2:StatusCode>90000</ns2:StatusCode><ns2:ErrorCode>EM1</ns2:ErrorCode><ns2:ErrorMsg>The SADAD Bills ID provided does not exist</ns2:ErrorMsg></BillQueryFault></detail></soap:Fault></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>e603bb46-3636-11ec-bf4a-0afc4a1d0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>217461255537</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>451.0</Amount><StatusCode>90005</StatusCode><ErrorMessage>Partial payments are not permitted</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>e0db96fa-339f-11ec-a054-0afc4a1e0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>890452823685</InvoiceCode><SupplierID>657</SupplierID><SupplierName>Wataniya insurance</SupplierName><Amount>1.0</Amount><StatusCode>90006</StatusCode><ErrorMessage>Amount entered is less than the minimum (partial payment) payable</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>0260f0c6-363a-11ec-bf4a-0afc4a1d0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>178027290132</InvoiceCode><SupplierID>512</SupplierID><SupplierName>Saudi Ceramic Company</SupplierName><Amount>3340.28</Amount><StatusCode>90007</StatusCode><ErrorMessage>Amount entered is greater than the invoice outstanding amount</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>4fea79da-34ee-11ec-a402-4b524c5eabc0</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>492369966504</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>649.83</Amount><StatusCode>90008</StatusCode><ErrorMessage>Payment failure, please try again</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>e603bb46-3636-11ec-bf4a-0afc4a1d0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>217461255537</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>451.0</Amount><StatusCode>90005</StatusCode><ErrorMessage>Partial payments are not permitted</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>e0db96fa-339f-11ec-a054-0afc4a1e0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>890452823685</InvoiceCode><SupplierID>657</SupplierID><SupplierName>Wataniya insurance</SupplierName><Amount>1.0</Amount><StatusCode>90006</StatusCode><ErrorMessage>Amount entered is less than the minimum (partial payment) payable</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>0260f0c6-363a-11ec-bf4a-0afc4a1d0000</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>178027290132</InvoiceCode><SupplierID>512</SupplierID><SupplierName>Saudi Ceramic Company</SupplierName><Amount>3340.28</Amount><StatusCode>90007</StatusCode><ErrorMessage>Amount entered is greater than the invoice outstanding amount</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/"><soap:Header><Header xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><ns2:RequestID>4fea79da-34ee-11ec-a402-4b524c5eabc0</ns2:RequestID></Header></soap:Header><soap:Body><PaymentValidationResponse xmlns:ns2="https://sih.sadad.com/common" xmlns="https://sih.sadad.com/PaymentValidation/defs"><StatusCode>10099</StatusCode><StatusDescription>Failure (see embedded error code(s))</StatusDescription><PaymentValidations><PaymentValidation><InvoiceCode>492369966504</InvoiceCode><SupplierID>536</SupplierID><SupplierName>Walaa Insurance Company</SupplierName><Amount>649.83</Amount><StatusCode>90008</StatusCode><ErrorMessage>Payment failure, please try again</ErrorMessage></PaymentValidation></PaymentValidations></PaymentValidationResponse></soap:Body></soap:Envelope>
<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">

Your configuration file probably. Is there only 1 record in that file or do similar messages repeat below it?

input {}
filter {}
output {}

similar messages repeat to test

Ok, thanks for the input. Do you want each one of those XML lines that begins with <soap:Envelope xmlns:soap to be it's own event? If that's the case then change your multiline pattern to:

  codec => multiline {
    pattern => '<soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/">'
    what => previous
    negate => true
   }