# Parse xml entry with logstash and create a new field with the data

**URL:** <https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262>\
**Category:** Logstash\
**Created:** [April 12, 2022, 5:32pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262 "2022-04-12T17:32:14Z")\
**Posts on this page:** 14\
**Page:** 2

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 20, 2022, 8:26pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/24 "2022-04-20T20:26:25Z")

</div>

I can see no reason why what you tried would not work.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 25, 2022, 5:36pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/25 "2022-04-25T17:36:18Z")

</div>

Hi @Badger ,

is there a way logstash read updated contents of a file before sending event to elastic.

In my build xml, there is a tag `<duration>`. it is assigned a value 0 in the start of the process(Jenkins job). and logstash reads the file and create event based on it. but once that Jenkins job is completed. the duration value get updated to the actual time taken by job to complete. but logstash doesn't update its value in the event and sends 0 as value in the event to elastic.

is there a way logstash can also read the updated value of tag which it already read?

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 25, 2022, 5:37pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/26 "2022-04-25T17:37:02Z")

</div>

Below is my logstash conf

```auto
 input {
      file {
        path => "/var/jenkins_home/jobs/CTSD/jobs/apis/jobs/eatviewer/branches/*/builds/*/build.xml"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        type => "xml"
        codec => multiline {
           pattern => '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2},[0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
           negate => true
           what => previous
           max_lines => 10000000000
           auto_flush_interval => 600
        }
      }
    }
    filter {
      xml {
        source => "message"
        store_xml => false
        xpath => [
            "/flow-build/startTime/text()", "startTime",
            "/flow-build/duration/text()", "duration",
            "/flow-build/execution/result/text()", "result"
        ]
      }
    }
    output {
      elasticsearch { hosts => ["https://elastic:443/elasticsearch"] index => "elktest-%{+YYYY.MM.dd}" }
      stdout { codec => rubydebug }
    }

```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 25, 2022, 5:41pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/27 "2022-04-25T17:41:25Z")

</div>

> [@Rohit\_Goel1](#):
>
> is there a way logstash can also read the updated value of tag which it already read?

If it rewrites the same file, then no, logstash cannot do that.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 26, 2022, 1:56pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/28 "2022-04-26T13:56:55Z")

</div>

Hi @Badger ,

thanks, but is there a way i read the xml file for parsing once `<duration>` value is non-zero in xml.  
I am currently stuck as without this my data being fetched from xml is not correct. only when duration has non-zero value my xml file will be valid to get parsed by logstash

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 26, 2022, 5:23pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/29 "2022-04-26T17:23:59Z")

</div>

Hi @Badger , can you please help suggest on this ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 26, 2022, 6:11pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/30 "2022-04-26T18:11:08Z")

</div>

You could use a conditional and `drop {}` the event if the value of the duration field is zero, or possibly if [message] contains \<duration\>0\</duration\>.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 26, 2022, 6:23pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/31 "2022-04-26T18:23:36Z")

</div>

Thanks @Badger , but will logstash read again this XML and generate an event once duration value is non zero ?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 26, 2022, 6:38pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/32 "2022-04-26T18:38:42Z")

</div>

As I said before, if the file gets rewritten then the file input will not reread it.

---

<div class="post-metadata">

**Author:** ![amsmzn](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/amsmzn/32/104115_2.png) [@amsmzn](https://discuss.elastic.co/u/amsmzn)\
**Post date:** [April 27, 2022, 3:20am UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/33 "2022-04-27T03:20:00Z")

</div>

is there a way, we can define in logstash.conf that once my file has been updated then logstash picks up the build.xml?

We tried with stat\_interval which would actually delay the reading of the build.xml by logstash but not sure if that would be an accurate solution.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 27, 2022, 3:41am UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/34 "2022-04-27T03:41:51Z")

</div>

The file input is designed to tail log files. If a new file is created (with a new inode) then logstash will read it. If the same file is rewritten from the beginning then a file input will not re-read it.

---

<div class="post-metadata">

**Author:** ![Rohit\_Goel1](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rohit_goel1/32/46637_2.png) [@Rohit\_Goel1](https://discuss.elastic.co/u/Rohit_Goel1)\
**Post date:** [April 27, 2022, 5:43pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/35 "2022-04-27T17:43:11Z")

</div>

Hi @Badger ,

I am trying to convert startTime field epoc value into UNIX time. but it is getting parse failure  
I want to convert the value of `<startTime>` and `<duration>` to readable format. Can you help what is the issue here ?

Below is my conf and as well result after logstash filteration:

````auto
 logstash.conf: |
    input {
      file {
        path => "/var/jenkins_home/jobs/CTSD/jobs/sdapplications/branches/master/builds/7/build.xml"
        start_position => "beginning"
        sincedb_path => "/dev/null"
        type => "xml"
        codec => multiline {
           pattern => '^[A-Z]{1}[a-z]{2} {1,2}[0-9]{1,2},[0-9]{4} {1,2}[0-9]{1,2}:[0-9]{2}:[0-9]{2}'
           negate => true
           what => previous
           max_lines => 10000000000
           auto_flush_interval => 60
        }
      }
    }
    filter {
      xml {
        source => "message"
        store_xml => false
        xpath => [
            "/flow-build/startTime/text()", "startTime",
            "/flow-build/duration/text()", "duration",
            "/flow-build/execution/result/text()", "result"
        ]
        remove_field => ["message"]
      }
    }
    filter {
      date {
      timezone => "UTC"
      match => ["startTime", "UNIX_MS"]
      target => "startTime"
      }
    }
    output {
      elasticsearch { hosts => ["https://elastic:443/elasticsearch"] index => "elktest-%{+YYYY.MM.dd}" }```

**Output :**

{
      "@version" => "1",
          "type" => "xml",
     "startTime" => [
        [0] "1651059323781"
    ],
          "path" => "/var/jenkins_home/jobs/CTSD/jobs/sdapplications/branches/master/builds/7/build.xml",
        "result" => [
        [0] "SUCCESS"
    ],
          "tags" => [
        [0] "multiline",
        [1] "_dateparsefailure"
    ],
          "host" => "elktest-0",
      "duration" => [
        [0] "423817"
    ],
    "@timestamp" => 2022-04-27T17:36:04.619Z
}
````

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 27, 2022, 5:52pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/36 "2022-04-27T17:52:13Z")

</div>

> [@Rohit\_Goel1](#):
>
> ```auto
> "startTime" => [
> [0] "1651059323781"
> ],
> 
> ```

startTime is an array, so you need `match => ["[startTime][0]", "UNIX_MS"]`

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 25, 2022, 5:52pm UTC](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262/37 "2022-05-25T17:52:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/parse-xml-entry-with-logstash-and-create-a-new-field-with-the-data/302262.md?page=1)