# Parse xml entry with logstash and create a new field with the data

**URL:** <https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262>\
**Category:** Logstash\
**Created:** [April 12, 2022, 5:32pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262 "2022-04-12T17:32:14Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 12, 2022, 5:32pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/1 "2022-04-12T17:32:14Z")

</div>

I need to parse below xml file with logstash and filter out status and create a new field with its value.

below is the sample xml file:

\<?xml version="1.1" encoding="UTF-8"?\>

\<flow="abc"\>  
`<tag>`SUCCESS`</tag>`  
`</flow>`

I have created below logstash but it is giving error and not giving required result

apiVersion: v1  
data:  
logstash.conf: |  
input {  
beats {  
port =\> 5044  
}  
}  
filter {  
xml {  
source =\> "message"  
target =\> "xml\_content"  
}  
split {  
field =\> "xml\_content[flow]"  
}  
split {  
field =\> "xml\_content[flow][result]"  
}  
mutate {  
add\_field =\> { "status" =\> "%{xml\_content[flow][result]}" }  
}  
}  
output { Elasticsearch { hosts =\> "Elasticsearch" } }

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 12, 2022, 5:37pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/2 "2022-04-12T17:37:18Z")

</div>

You need to format your post using markdown. If you Google "markdown tutorial" you will find multiple sites that provide one. Use the preview pane on the right of the edit pane to make sure the code is formatted correctly.

What error do you get? What does the [message] field of your event look like? (Expand an event in the Discover pane of Kibana and copy and paste from the JSON tab.)

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 12, 2022, 5:50pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/3 "2022-04-12T17:50:29Z")

</div>

Hello Badger ,

message field contains the complete xml

\<?xml version="1.1" encoding="UTF-8"?\>

`<flow-build plugin="workflow-job@1145.v7f2433caa07f">`  
`<actions>`  
`<hudson.model.CauseAction>`  
`<causeBag class="linked-hash-map">`  
`<entry>`  
`<hudson.model.Cause_-UserIdCause>`  
`<userId>503260426</userId>`  
`</hudson.model.Cause_-UserIdCause>`  
`<int>1</int>`  
`</entry>`  
`<queueId>73</queueId>`  
`<timestamp>1649770295128</timestamp>`  
`<startTime>1649770295158</startTime>`  
`<result>SUCCESS</result>`  
`</flow-build>`

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 12, 2022, 5:52pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/4 "2022-04-12T17:52:29Z")

</div>

I want to fetch result data whether SUCCESS or FAILURE and create a new filed for it

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 12, 2022, 5:55pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/5 "2022-04-12T17:55:04Z")

</div>

@Badger can you please suggest

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 12, 2022, 6:44pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/6 "2022-04-12T18:44:41Z")

</div>

If the [message] field contains that text then

```
xml {
    source => "message"
    store_xml => false
    xpath => { "//result/text()" => "result" }
}

```

will produce

```
    "result" => [
    [0] "SUCCESS"
],

```

using xpath _always_ results in the extracted objects being arrays. You can adjust that using

```
mutate { replace => { "result" => "%{[result][0]}" } }

```

to get

```
    "result" => "SUCCESS",

```

If you edit the XML to be valid (by closing the actions, hudson.model.CauseAction, and causeBag elements) then another way to do it would be

```
    xml {
        source => "message"
        target => "theXML"
        force_array => false
    }

```

which gets you

```
                      "result" => "SUCCESS",
                     "queueId" => "73",
                       "entry" => {
        "hudson.model.Cause_-UserIdCause" => {
            "userId" => "503260426"
        },

```

etc,

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 13, 2022, 7:19am UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/7 "2022-04-13T07:19:18Z")

</div>

Hello @Badger ,

I tried below as you suggested. Although it is adding new field but value in field is not getting replaced

filter {  
xml {  
source =\> "message"  
store\_xml =\> false  
xpath =\> { "//flow-build/result/text()" =\> "result" }  
}  
mutate {  
replace =\> { "result" =\> "%{result}" }  
}  
}

Field in kibana:

![image](https://us1.discourse-cdn.com/elastic/original/3X/c/2/c28e522f6583380c49651bc538dd8351ed9a3373.png)

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 13, 2022, 7:23am UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/8 "2022-04-13T07:23:07Z")

</div>

I tied both with array and without array

![image](https://us1.discourse-cdn.com/elastic/original/3X/6/1/615d208c2cd2788a37b433bcaedc7b45a24e37f9.png)

I tried below logstash.conf as well

logstash.conf: |  
input {  
beats {  
port =\> 5044  
}  
}  
filter {  
prune {  
blacklist\_names =\> [" \e[8mha._\e[0m "]  
}  
}  
filter {  
mutate {  
gsub =\> ["message", "\e[8mha._\e[0m", ""]  
}  
}  
filter {  
xml {  
source =\> "message"  
store\_xml =\> false  
xpath =\> { "/flow-build[@plugin]/result/text()" =\> "result" }  
}  
mutate {  
add\_field =\> { "result" =\> "%{result}" }  
}  
}  
output { Elasticsearch { hosts =\> "Elasticsearch:9200" } }

to fetch the string in result tag in xml under structure /flow-build/result to create a new field and display in kibana

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 13, 2022, 6:00pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/9 "2022-04-13T18:00:54Z")

</div>

As I said, _if_ the [message] field contains the text you said it contains then the filter configuration I posted will work. If it contains something else then it may not. Given that what you posted is not valid XML I suspect the [message] field may be slightly different.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 14, 2022, 3:06am UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/10 "2022-04-14T03:06:40Z")

</div>

Hello @Badger , there is an XML in the message field in kibana. Is there any way to confirm the same ?  
I am new to logstash and kibana. Can you please let me know if any way to check what is in message field

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 14, 2022, 4:05am UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/11 "2022-04-14T04:05:03Z")

</div>

Earlier today (my today, possibly your yesterday) you started another thread, which I spent some time working on, and then you deleted it before I could post my answer.

An XML filter expects a single XML element to surround everything in the source field. If there are two or more top-level XML elements then it will complain about trying to add a second item at the root.

You may be able to fix the message using something like

```
mutate { gsub => ["sourceField", "</endOfUsefulPart>.*", "</endOfUsefulPart>"] }

```

And I realize that there was a ton of information in that post that you probably did not want to share. But it is hard for us to help you without a reproducible failure. If you do provide one then I, and several other folks, will be happy to test it and help.

Spending time to narrow down a reproducible example adds a skill that will let you get more answers from more people.

As an example of reproduction... if you have a grok pattern that include IPV4, do not obfuscate your IP address as "a.b.c.d" (which is not a valid IP address), just replace it with "1.2.3.4" (which _is_ valid). It is a trivial change for you and makes testing things easier for every person who reviews questions here, and _much_ more likely that one of us will take the time to provide guidance.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 14, 2022, 6:59pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/13 "2022-04-14T18:59:19Z")

</div>

If your message field contains that complete XML flow-build element then

```
    xml {
        source => "message"
        store_xml => false
        xpath => { "/flow-build/result/text()" => "result" }
        remove_field => ["message"]
    }

```

will produce

```
    "result" => [
    [0] "SUCCESS"
]
```

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 14, 2022, 7:06pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/14 "2022-04-14T19:06:31Z")

</div>

Hello @Badger ,

I tried above but it did not create new field in document in kibana with name result when xml was parsed by logstash and sent to Elasticsearch.  
do i need to add something else to put the value fetched by xpath and create a new field out of it ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 14, 2022, 7:18pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/15 "2022-04-14T19:18:03Z")

</div>

> [@Rohit\_Goel1](#):
>
> I tried above but it did not create new field in document

I do not know why it would not do so.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 15, 2022, 7:32pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/16 "2022-04-15T19:32:27Z")

</div>

Hello @Badger ,

It seems like my filebeat which is sending source xml file(build.xml) in message field to logstash is getting corrupted while being transfered. I tried to test both xml from source location and message field. source location xml file is in correct format. but when it is sent in message field it's structure is getting changed.  
Do you know any way to resolve this ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 15, 2022, 7:45pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/17 "2022-04-15T19:45:35Z")

</div>

That sounds like a filebeat question, and one I cannot answer.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 16, 2022, 4:49pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/18 "2022-04-16T16:49:10Z")

</div>

thanks @Badger, I found some part of xml is getting repeated which in message field like below which is causing the issue while parsing the xml

below part is coming after end tag `</flow-build>`

Is there a way to mark the beginning and end tag of xml in logstash to be parsed ?

```auto
 <name>master</name>
          </hudson.plugins.git.BranchSpec>
        </branches>
        <doGenerateSubmoduleConfigurations>false</doGenerateSubmoduleConfigurations>
        <submoduleCfg class="empty-list"/>
        <extensions>
          <jenkins.plugins.git.GitSCMSourceDefaults>
            <includeTags>false</includeTags>
          </jenkins.plugins.git.GitSCMSourceDefaults>
          <hudson.plugins.git.extensions.impl.BuildChooserSetting>
            <buildChooser class="jenkins.plugins.git.AbstractGitSCMSource$SpecificRevisionBuildChooser">
              <revision reference="../../../../../../../actions/hudson.plugins.git.util.BuildData[3]/buildsByBranchName/entry/hudson.plugins.git.util.Build/marked"/>
            </buildChooser>
          </hudson.plugins.git.extensions.impl.BuildChooserSetting>
        </extensions>
      </scm>
      <node>build-agent-n8m6b</node>
      <workspace>/home/jenkins/agent/workspace/CTSD_apis_eatviewer_master</workspace>
      <pollingBaseline class="hudson.scm.SCMRevisionState$None" reference="../../../actions/org.jenkinsci.plugins.workflow.steps.scm.MultiSCMRevisionState/revisionStates/entry/hudson.scm.SCMRevisionState_-None"/>
    </org.jenkinsci.plugins.workflow.job.WorkflowRun_-SCMCheckout>
  </checkouts>

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 16, 2022, 4:58pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/19 "2022-04-16T16:58:21Z")

</div>

If you want to discard that you could try

```
mutate { gsub => ["message", "(</flow-build>).*", "\1"] }

```

which will keep the `</flow-build>` but delete everything after it.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 18, 2022, 12:56pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/21 "2022-04-18T12:56:58Z")

</div>

thanks @Badger . I changed and added below then it worked  
I was able to get `<result>` as a field

```auto
 filter {
      mutate {
        gsub => ["message", "(</flow-build>).*</checkouts>", "\1"]
      }
    }

```

I also want `<startTime>` to be extracted from xml in "message" field which has epoc time as value and create a new field for it as well after converting epoc to UNIX time

I tried below code to extract both fields, but it just created `<startTime>` field and not result field

```auto
data:
  logstash.conf: |
    input {
      beats {
        port => 5044
      }
    }
    filter {
      mutate {
        gsub => ["message", "(</flow-build>).*</checkouts>", "\1"]
      }
    }
    filter {
      xml {
        source => "message"
        store_xml => false
        xpath => [
           "/flow-build/result/text()", "result",
           "/flow-build/startTime/text()", "startTime"
        ]
        remove_field => ["message"]
      }
    }

```

Any way if i can extract multiple fields from xml and the remove message field ?

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 20, 2022, 8:20pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/23 "2022-04-20T20:20:47Z")

</div>

Hello @Badger , can you please suggest me here if possible ?  
Thanks for your support

[Next page](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262.md?page=2)