Help parse xml from logs file

[Nurlan199206]

Help pls parse xml data from log files.

i'm trying use xml filter plugin

my conf file

filter {
  if "xml" in [tags] {
  xml {
    namespaces => {
             "soapenv" => "http://schemas.xmlsoap.org/soap/envelope/"
             "ns1" => "http://www.w3.org/2001/XMLSchema-instance"
             "ns2" => "urn:kz:company:int:header"
                }
    source => "message"
    store_xml => "false"
    xpath => [ "<soap:Envelope/EventId/text()", "EventId" ]
    }
}

xml data from logs looks like that.

<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><soapenv:Header><ns1:Header xmlns:ns1="urn:kz:company:int:header"><EventId>e293f776-e7d2-4063-89a3-080a0d39da87</EventId><Initiator>BPM.12194683</Initiator><Source>BPM.RegisterCase</Source></ns1:Header></soapenv:Header><soapenv:Body><ns3:SaveActivityReq xmlns:ns3="urn:kz:company:int:casemanagementmessage"><CrmUid>8d4ba904-90f3-4630-bb5e-3a32b4296708</CrmUid><CrmProcessId>12196592</CrmProcessId><SrList><CrmSr><Issue>Первичная регистрация</Issue><SubIssue>Первичная регистрация</SubIssue><Status>Error</Status><Comment>ERROR FINISH PROCESS(TERMINATED_BY_TIMER)</Comment><ChildSrList><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>Check Active</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>check age by IIN</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>CheckClientExistInColvir</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>check tax debt</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>GetClientInfoByIin</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>DocForLivenessForm</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>1 DBO Notification</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>Check Not Digital</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>Check Activi IIN</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>Check Juridical</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>setFATCAflag</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>ECM:CreateFATCA</SubIssue><Status>Done</Status><Comment></Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr><ChildCrmSr><Issue>Первичная регистрация</Issue><SubIssue>ERROR FINISH PROCESS</SubIssue><Status>Error</Status><Comment>TERMINATED_BY_TIMER</Comment><BpmId>12196592</BpmId><BpmDate>11/29/2020</BpmDate></ChildCrmSr></ChildSrList></CrmSr></SrList></ns3:SaveActivityReq></soapenv:Body></soapenv:Envelope>

error log:

[2020-11-29T21:40:44,517][DEBUG][logstash.filters.xml     ][main][d15609a5c74df9c8c58c39f92f2af5fd2e770ec782e515213649e172701306c8] Running xml filter {:event=>#<LogStash::Event:0x6788bb58>}
[2020-11-29T21:40:44,710][DEBUG][logstash.filters.xml     ][main][d15609a5c74df9c8c58c39f92f2af5fd2e770ec782e515213649e172701306c8] Parsed xml with 2 errors
[2020-11-29T21:40:44,776][WARN ][logstash.filters.xml     ][main][d15609a5c74df9c8c58c39f92f2af5fd2e770ec782e515213649e172701306c8] XML Parse Error {:exception=>"<soap:Envelope/urn:kz:company:int:header/EventId/text()", :source=>"message"

[Badger]

    xml {
        namespaces => {
            "soapenv" => "http://schemas.xmlsoap.org/soap/envelope/"
            "ns3" => "urn:kz:company:int:casemanagementmessage"
            "ns1" => "urn:kz:company:int:header"
        }
        source => "message"
        xpath => { "/soapenv:Envelope/soapenv:Header/ns1:Header/EventId/text()" => "EventId" }
    }

gets me

   "EventId" => "e293f776-e7d2-4063-89a3-080a0d39da87",

Note that I changed ns2 to ns3 and changed the values of both ns1 and ns3 to match the values in the XML.

[Nurlan199206]

Thank you for reply.

it's weird. Pipeline worked without any errors, but i don't see new field "EventId" on left panel.

[2020-11-30T11:22:41,596][DEBUG][org.logstash.beats.BeatsHandler][main][c09623f54506241d5336dd5e9474106d26f465e56d18179d265fc077484e2c0b] b991bd36: batches pending: false
[2020-11-30T11:22:41,710][DEBUG][logstash.filters.xml     ][main][8362fbf02e086945e600ccdffdd33d63d68e9d00478a7a0fb8bcfacb6edbf2c0] Running xml filter {:event=>#<LogStash::Event:0x21fe4925>}
[2020-11-30T11:22:42,074][DEBUG][logstash.filters.xml     ][main][8362fbf02e086945e600ccdffdd33d63d68e9d00478a7a0fb8bcfacb6edbf2c0] Event after xml filter {:event=>#<LogStash::Event:0x21fe4925>}