# Need help to parse xml file with logstash

**URL:** https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464
**Category:** Logstash
**Created:** [August 23, 2019, 8:03am UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464 "2019-08-23T08:03:00Z")
**Posts on this page:** 13
**Page:** 1

<div class="post-metadata">

### Author: ![ROok](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@ROok](https://discuss.elastic.co/u/ROok)
#### Post date: [August 23, 2019, 8:03am UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/1 "2019-08-23T08:03:00Z")

</div>

Hi All,

I need help for parsing my xml file  
My xml file look like this:

```
 <UserList>
    <User statefile='10'>
       <UserAt atName='NUMBER'>3312348555</UserAt>
       <UserAt atName='ID'>308014204145695</UserAt>
       <UserAt atName='segTo'>X5XXX</UserAt>
       <UserAt atName='descriptionSegTo'>XXX 10Ci 4C XX XXX</UserAt >
       <UserAt atName='payMode'>yes</UserAt >
          </User>
   <User statefile='60'>
       <UserAt atName='NUMBER'>3312348555</UserAt>
       <UserAt atName='ID'>308014204145695</UserAt>
       <UserAt atName='segTo'>X5XXX</UserAt>
       <UserAt atName='descriptionSegTo'>XXX 10Ci 4C XX XXX</UserAt >
       <UserAt atName='payMode'>yes</UserAt>
       <ServPrest hash='TTGHB' date=''/>
  </User>
 ..... 
 </UserList>

```

I want my output in this format:

```
"NUMBER" => 3312348555
 "ID" => 308014204145695
 "segTo" => X5XXX
 "descriptionSegTo" => XXX 10Ci 4C XX XXX
 "payMode" => yes

"NUMBER" => 3312348555
 "ID" => 308014204145695
 "segTo" => X5XXX
 "descriptionSegTo" => XXX 10Ci 4C XX XXX
 "payMode" => yes
 "hash" => TTGHB 
 .....

```

My logstash configuration:

```
input {
    file {
       path => "/home/osad/fichierxml"
       sincedb_path => "/dev/null"
       start_position => "beginning"
       codec => multiline {
          pattern => "<User>"
          negate => true
          what => "previous"
      }
    }
 }

filter {
    xml{
       source => "message"
       store_xml => false
      # target => "doc"
       xpath =>
        [
          "/User/UserAt/@atName='NUMBER'","NUMBER",
          "/User/UserAt/@atName='ID'","ID",
          "/User/UserAt/@atName='segTo'","SEGTO",
  ] 

 }
   }

output {
  elasticsearch {
      hosts => ["http://localhost:9200"]
      index => "test_xml"
    }
    stdout {
     codec => rubydebug
    }
}

```

But I have any results when I run logstash. it stays stuck on:

```
 [2019-08-23T09:29:17,519][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"main"}
[2019-08-23T09:29:17,812][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-08-23T09:29:17,910][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections
[2019-08-23T09:29:19,373][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}

```

Thank you in advance !

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 23, 2019, 1:57pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/2 "2019-08-23T13:57:16Z")

</div>

> [@ROok](#):
>
> pattern =\> "\<User\>"

That does not match any of the lines in your file. You could try removing the trailing \>

---

<div class="post-metadata">

### Author: ![ROok](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@ROok](https://discuss.elastic.co/u/ROok)
#### Post date: [August 23, 2019, 2:27pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/3 "2019-08-23T14:27:16Z")

</div>

Hi @Badger  
Thanks for your reply

I did what you suggested but I always have the same result  
Is my xpath correct ?

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 23, 2019, 2:55pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/4 "2019-08-23T14:55:50Z")

</div>

If you use

```
pattern => "<User "

```

with a trailing space the xpath appears to parse.

```
    "NUMBER" => [
    [0] "true"
],
     "SEGTO" => [
    [0] "true"
],
        "ID" => [
    [0] "true"
],

```

Not sure whether that's what you wanted though.

---

<div class="post-metadata">

### Author: ![ROok](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@ROok](https://discuss.elastic.co/u/ROok)
#### Post date: [August 23, 2019, 3:16pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/5 "2019-08-23T15:16:15Z")

</div>

I don't really understand what you means.  
Is that what you asked me:

```
input {
    file {
       path => "/home/osad/fichierxml"
       sincedb_path => "/dev/null"
       start_position => "beginning"
       codec => multiline {
          pattern =>"</User>"
          negate => true
          what => "previous"
      }
    }
 }

filter {
    xml{
       source => "message"
       store_xml => false
      # target => "doc"
       xpath =>[ "/User/UserAt/@atName='NUMBER'","NUMBER",
                  "/User/UserAt/@atName='ID'","ID",
                  "/User/UserAt/@atName='segTo'","SEGTO"
                 ]
 }
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 23, 2019, 3:29pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/6 "2019-08-23T15:29:31Z")

</div>

> [@ROok](#):
>
> codec =\> multiline { pattern =\>"\</User\>" negate =\> true what =\> "previous" }

That's not going to work. I suggest you add

```
output { stdout { codec => rubydebug {} } }

```

and focus on getting the multiline codec working before you worry about the xpath expressions.

---

<div class="post-metadata">

### Author: ![ROok](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@ROok](https://discuss.elastic.co/u/ROok)
#### Post date: [August 23, 2019, 4:29pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/7 "2019-08-23T16:29:19Z")

</div>

I changed the multine codec to this:

```
    codec => multiline {
             pattern => "^"
             negate => true
             what => "previous"

```

But the result is still blocked on:

```
[2019-08-23T18:25:41,126][INFO][logstash.javapipeline] Pipeline started {"pipeline.id"=>"main"}
[2019-08-23T18:25:41,318][INFO][logstash.agent] Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[2019-08-23T18:25:41,429][INFO][filewatch.observingtail] START, creating Discoverer, Watch with file and sincedb collections
[2019-08-23T18:25:42,722][INFO][logstash.agent] Successfully started Logstash API endpoint {:port=>9600}
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 23, 2019, 6:08pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/8 "2019-08-23T18:08:43Z")

</div>

> [@ROok](#):
>
> codec =\> multiline { pattern =\> "^" negate =\> true what =\> "previous"

That pattern matches every line, so this will never combine lines. If you change negate =\> false then it will combine every line in the file into a single event. However, it will never flush it to the pipeline unless you add 'auto\_flush\_interval =\> 1'

You could then parse the XML using this

```
    xml{
        source => "message"
        store_xml => true
        target => "theXML"
        force_array => false
        remove_field => ["message"]
    }
    split { field => "[theXML][User]" }

```

which will get you events that look like this

```
    "theXML" => {
    "User" => {
           "UserAt" => [
            [0] {
                 "atName" => "NUMBER",
                "content" => "3312348555"
            },
            [1] {
                 "atName" => "ID",
                "content" => "308014204145695"
            },
            [2] {
                 "atName" => "segTo",
                "content" => "X5XXX"
            },
            [3] {
                 "atName" => "descriptionSegTo",
                "content" => "XXX 10Ci 4C XX XXX"
            },
            [4] {
                 "atName" => "payMode",
                "content" => "yes"
            }
        ],
        "statefile" => "60",
        "ServPrest" => {
            "date" => "",
            "hash" => "TTGHB"
        }
    }
},

```

You might then want to do something like

```
    ruby {
        code => '
            u = event.get("[theXML][User][UserAt]")
            if u
                h = {}
                u.each_index { |x|
                    h[u[x]["atName"] ] = u[x]["content"]
                }
                event.set("[theXML][User][UserAt]", h)
            end
        '
    }

```

which would change that to

```
    "theXML" => {
    "User" => {
           "UserAt" => {
            "descriptionSegTo" => "XXX 10Ci 4C XX XXX",
                     "payMode" => "yes",
                       "segTo" => "X5XXX",
                          "ID" => "308014204145695",
                      "NUMBER" => "3312348555"
        },
        "statefile" => "60",
        "ServPrest" => {
            "date" => "",
            "hash" => "TTGHB"
        }
    }
},

```

---

<div class="post-metadata">

### Author: ![ROok](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@ROok](https://discuss.elastic.co/u/ROok)
#### Post date: [August 26, 2019, 8:55am UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/9 "2019-08-26T08:55:29Z")

</div>

Hi @Badger  
Thanks for your reply  
I have tested your solution but I don't have the same result as you do.  
I have some error like:

[2019-08-26T10:48:46,039][WARN][logstash.filters.xml] Error parsing xml with XmlSimple {:source=\>"message", :value=\>" ", :exception=\>#\<REXML::ParseException: No close tag for /User

[2019-08-26T10:48:46,114][WARN][logstash.filters.split] Only String and Array types are splittable. field:[theXML][User] is of type = NilClass  
[2019-08-26T10:48:46,115][WARN][logstash.filters.split] Only String and Array types are splittable. field:[theXML][User] is of type = NilClass

"tags" =\> [  
[0] "\_xmlparsefailure",  
[1] "\_split\_type\_failure"

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 26, 2019, 12:35pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/10 "2019-08-26T12:35:03Z")

</div>

> [@ROok](#):
>
> [2019-08-26T10:48:46,039][WARN][logstash.filters.xml] Error parsing xml with XmlSimple {:source=\>"message", :value=\>" ", :exception=\>#\<REXML::ParseException: No close tag for /User

Use

```
output { stdout { codec => rubydebug } }

```

and verify that the event produced by the multiline codec is valid XML. If it is not then the xml filter will not parse it, and then the other filters will not work.

---

<div class="post-metadata">

### Author: ![ROok](https://avatars.discourse-cdn.com/v4/letter/r/57b2e6/32.png) [@ROok](https://discuss.elastic.co/u/ROok)
#### Post date: [August 26, 2019, 2:17pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/11 "2019-08-26T14:17:53Z")

</div>

I don't have a multiline error in tags but I have the impression that multiline doesn't work.  
The result look like this:

```
   "tags" => [
        [0] "_xmlparsefailure",
        [1] "_split_type_failure"
    ],
          "path" => "/home/osad/fichierxml",
    "@timestamp" => 2019-08-26T13:24:17.174Z,
          "host" => "elk.lab.fr",
      "@version" => "1"
}
{
          "tags" => [
        [0] "_split_type_failure"
    ],
          "user" => {
         "atName" => "NUMBER",
        "content" => "3312348555"
    },
          "path" => "/home/osad/fichierxml",
    "@timestamp" => 2019-08-26T13:24:17.174Z,
          "host" => "elk.lab.fr",
      "@version" => "1"
}
{
          "tags" => [
        [0] "_split_type_failure"
    ],
          "user" => {
         "atName" => "ID",
        "content" => "308014204145695"
    },
          "path" => "/home/osad/fichierxml",
    "@timestamp" => 2019-08-26T13:24:17.175Z,
          "host" => "elk.lab.fr",
      "@version" => "1"
}
{
          "tags" => [
        [0] "_split_type_failure"
    ],
          "user" => {
         "atName" => "segTo",
        "content" => "X5XXX"
    },
          "path" => "/home/osad/fichierxml",
    "@timestamp" => 2019-08-26T13:24:17.176Z,
          "host" => "elk.lab.fr",
      "@version" => "1"
}
```

---

<div class="post-metadata">

### Author: ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)
#### Post date: [August 26, 2019, 2:30pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/12 "2019-08-26T14:30:10Z")

</div>

If you have multiple events and a \_split\_type\_failure tag then we can be sure that the multiline filter did not combine the entire file into a single event. That's what you need to fix first.

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [September 23, 2019, 2:30pm UTC](https://discuss.elastic.co/t/need-help-to-parse-xml-file-with-logstash/196464/13 "2019-09-23T14:30:11Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.