# Help with parsing xml data

**URL:** <https://discuss.elastic.co/t/help-with-parsing-xml-data/127383>\
**Category:** Logstash\
**Created:** [April 9, 2018, 8:23pm UTC](https://discuss.elastic.co/t/help-with-parsing-xml-data/127383 "2018-04-09T20:23:51Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![kattula](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@kattula](https://discuss.elastic.co/u/kattula)\
**Post date:** [April 9, 2018, 8:23pm UTC](https://discuss.elastic.co/t/help-with-parsing-xml-data/127383/1 "2018-04-09T20:23:51Z")

</div>

Hi everyone,

I'm new to using ELK stack. I'm trying to parse an xml audit file. Even though I get the output as I need, because of the length of code I'm assuming there could be a simpler solution than what I've written. The XML filter doesn't just give values, instead I'm getting the tags along with it which I'm removing with gsub. Can someone please help me with this?

Thanks,  
Krishna

**Sample of data I'm using:**

```
<engineAuditEvent encrypted="n" signed="n">
    <localTime>2018-03-09 16:25:21.520 -0500</localTime>
    <time>2018-03-09 21:25:21.520</time>
    <utc>2018-03-09 21:25:21.520</utc>
    <engineName>TESTENG1</engineName>
    <msg>
        <msgID>AUD0205I</msgID>
        <msgText>Session 34f32f38-12fd-4907-abfe-3036cd328847 [SFTP]: Outbound plain socket connection with primary node ftpp.test established.</msgText>
        <condCode>0</condCode>
        <auditData>
            <sessionId>34f32f38-12fd-4907-abfe-3036cd328847</sessionId>
            <protocol>SFTP</protocol>
            <clientAddress>8.8.8.8</clientAddress>
            <clientPort>46723</clientPort>
            <clientIdent>
                <userid>TEST</userid>
            </clientIdent>
            <serverAddress>ftpp.test</serverAddress>
            <serverPort>10022</serverPort>
            <outboundPort>34203</outboundPort>
            <serverIdent>
                <userid>TEST</userid>
            </serverIdent>
        </auditData>
    </msg>
</engineAuditEvent>

```

**test-pipeline.conf**

```
input {
 file {
  path => "/home/testdev/ELK/test-logs/testlog.xml"
  start_position => "beginning"
  sincedb_path => "/dev/null"
  type => xml
	codec => multiline {
		pattern => '</engineAuditEvent>'
		negate => true
		what => next
		}   
	}
}

# ---------------- Filter ----------------
filter {
if [type] == "xml" {
 
	xml {
# remove_namespaces => true
		store_xml => "false" 
		source => "message"
		xpath => [ 
			"/engineAuditEvent/localTime", "est",
			"/engineAuditEvent/msg/msgText", "msg",
			"/engineAuditEvent/msg/msgText", "msgID",
			"/engineAuditEvent/engineName", "engine",			
			"/engineAuditEvent/msg/auditData/sessionId", "sessionid",
			"/engineAuditEvent/msg/auditData/sessionId", "clientAddress",
			"/engineAuditEvent/msg/auditData/clientIdent/userid", "userid" 
			]
		}
	mutate {
     		gsub => ["est", "<localTime>", ""]
			gsub => ["msg", "<msgText>", ""]
			gsub => ["msg", "</msgText>", ""]
			gsub => ["sessionid", "<sessionId>", ""]
			gsub => ["sessionid", "</sessionId>", ""]
			gsub => ["userid", "<userid>", ""]
			gsub => ["userid", "</userid>", ""]
			gsub => ["msgID", "<msgid>", ""]
			gsub => ["msgID", "</msgid>", ""]
			gsub => ["clientAddress", "<clientip>", ""]
			gsub => ["clientAddress", "</clientip>", ""]
		}
	grok {
			match => { "est" => "%{TIMESTAMP_ISO8601:localtime}%{GREEDYDATA:extradata}" }
			remove_field => ["extradata", "est"]			
			}
	date {
			match => ["localtime", "yyyy-MM-dd HH:mm:ss.SSS"]
			target => "@timestamp"
          } 
		  
	grok {
			match => { "msg" => "%{GREEDYDATA:msgtxt}" }
# remove_field => ["msg", "tags", "message"]			
	}
	grok {
			match => { "userid" => "%{GREEDYDATA:loginid}" }	
	}
	grok {
			match => { "sessionid" => "%{GREEDYDATA:session-id}" }			
	}
		
	if "_grokparsefailure" in [tags] {
                drop { }
        }

 }
}

output {
	stdout { codec => rubydebug }
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 9, 2018, 9:06pm UTC](https://discuss.elastic.co/t/help-with-parsing-xml-data/127383/2 "2018-04-09T21:06:40Z")

</div>

OK. There are better xpath expressions that you can use. Instead of fetching the XML node, using /engineAuditEvent/localTime, you should pull the text out of each node. Something like this will work better.

```auto
      xpath => {
        "/engineAuditEvent/localTime/text()" => "est"
        "/engineAuditEvent/msg/msgText/text()" => "msg"
        "/engineAuditEvent/msg/msgID/text()" => "msgID"
        "/engineAuditEvent/engineName/text()" => "engine"
        "/engineAuditEvent/msg/auditData/sessionId/text()" => "sessionid"
        "/engineAuditEvent/msg/auditData/clientAddress/text()" => "clientAddress"
        "/engineAuditEvent/msg/auditData/clientIdent/userid/text()" => "userid"
      }

```

The next problem you will have is that xpath ignores force\_array =\> false, and still puts everything into arrays. So you then have to do this to them all.

```auto
if [fieldx] { mutate { replace => { "fieldx" => "%{[fieldx][0]}" } } }

```

You may find it easier to set store\_xml =\> true, and add target =\> theXML, then mutate+add\_field to pull stuff out of theXML, and then delete that field.

BTW, mutate+add\_field or mutate+rename would be a more traditional approach than

```auto
	grok {
			match => { "sessionid" => "%{GREEDYDATA:session-id}" }			
	}

```

---

<div class="post-metadata">

**Author:** ![kattula](https://avatars.discourse-cdn.com/v4/letter/k/f19dbf/32.png) [@kattula](https://discuss.elastic.co/u/kattula)\
**Post date:** [April 9, 2018, 11:50pm UTC](https://discuss.elastic.co/t/help-with-parsing-xml-data/127383/3 "2018-04-09T23:50:01Z")

</div>

Thank you very much Badger.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 7, 2018, 11:50pm UTC](https://discuss.elastic.co/t/help-with-parsing-xml-data/127383/4 "2018-05-07T23:50:34Z")

</div>

Closed on May 7, 2018, 11:50pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.