# Logstash XML file parsing - awkward Splitting event Problem

**URL:** <https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906>\
**Category:** Logstash\
**Created:** [April 2, 2019, 5:03am UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906 "2019-04-02T05:03:31Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![NativAt](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@NativAt](https://discuss.elastic.co/u/NativAt)\
**Post date:** [April 2, 2019, 5:03am UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/1 "2019-04-02T05:03:31Z")

</div>

I'm working with ELK 6.7.0 on docker with official images. This is my conf file:

```
input {
  file {
    path => "/usr/share/logstash/logs/*.xml"
    type => "xml"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "<root>"
      negate => "true"
      what => "previous"
    }
  }
}

filter {  
  xml {
    source => "message"
    store_xml => false
    xpath => [
        "/root/ChainId/text()", "ChainId",
        "/root/SubChainId/text()", "SubChainId",
        "/root/StoreId/text()", "StoreId",
        "/root/BikoretNo/text()", "BikoretNo",
        "/root/DllVerNo/text()", "DllVerNo"
    ]
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "xml_index"
  }

  stdout { 
    codec => rubydebug 
  }
}

```

My XML file is:

```
<?xml version="1.0" encoding="UTF-8"?>
<root>
    <ChainId>7290027600007</ChainId>
    <SubChainId>001</SubChainId>
    <StoreId>001</StoreId>
    <BikoretNo>9</BikoretNo>
    <DllVerNo>8.0.1.3</DllVerNo>
</root>

```

I'm trying to parse incoming XML files, but when a new file is created on the path folder logstash parsing it as following:

```
logstash_1 | {
logstash_1 | "path" => "/usr/share/logstash/logs/example10.xml",
logstash_1 | "@version" => "1",
logstash_1 | "message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>",
logstash_1 | "type" => "xml",
logstash_1 | "@timestamp" => 2019-04-02T04:42:59.248Z,
logstash_1 | "host" => "a4f1bf64a3d5"
logstash_1 | }
```

However, When I reload my conf file Logstash surprisingly is parsing my XML successfully:

```
logstash_1 | {
logstash_1 | "StoreId" => [
logstash_1 | [0] "001"
logstash_1 | ],
logstash_1 | "message" => "<root>\n <ChainId>7290027600007</ChainId>\n <SubChainId>001</SubChainId>\n <StoreId>001</StoreId>\n <BikoretNo>9</BikoretNo>\n <DllVerNo>8.0.1.3</DllVerNo>",
logstash_1 | "DllVerNo" => [
logstash_1 | [0] "8.0.1.3"
logstash_1 | ],
logstash_1 | "type" => "xml",
logstash_1 | "SubChainId" => [
logstash_1 | [0] "001"
logstash_1 | ],
logstash_1 | "BikoretNo" => [
logstash_1 | [0] "9"
logstash_1 | ],
logstash_1 | "path" => "/usr/share/logstash/logs/example10.xml",
logstash_1 | "@version" => "1",
logstash_1 | "ChainId" => [
logstash_1 | [0] "7290027600007"
logstash_1 | ],
logstash_1 | "tags" => [
logstash_1 | [0] "multiline"
logstash_1 | ],
logstash_1 | "@timestamp" => 2019-04-02T04:43:18.439Z,
logstash_1 | "host" => "a4f1bf64a3d5"
logstash_1 | }
logstash_1 | {
logstash_1 | "StoreId" => [
logstash_1 | [0] "001"
logstash_1 | ],
logstash_1 | "message" => "<root>\n <ChainId>7290027600007</ChainId>\n <SubChainId>001</SubChainId>\n <StoreId>001</StoreId>\n <BikoretNo>9</BikoretNo>\n <DllVerNo>8.0.1.3</DllVerNo>",
logstash_1 | "DllVerNo" => [
logstash_1 | [0] "8.0.1.3"
logstash_1 | ],
logstash_1 | "type" => "xml",
logstash_1 | "SubChainId" => [
logstash_1 | [0] "001"
logstash_1 | ],
logstash_1 | "BikoretNo" => [
logstash_1 | [0] "9"
logstash_1 | ],
logstash_1 | "path" => "/usr/share/logstash/logs/example11.xml",
logstash_1 | "@version" => "1",
logstash_1 | "ChainId" => [
logstash_1 | [0] "7290027600007"
logstash_1 | ],
logstash_1 | "tags" => [
logstash_1 | [0] "multiline"
logstash_1 | ],
logstash_1 | "@timestamp" => 2019-04-02T04:43:18.440Z,
logstash_1 | "host" => "a4f1bf64a3d5"
logstash_1 | }
```

The message field in both events is different parts of the file and seems like Logstash is splitting the file before and after the pattern. Even so, not clear why it doing it just on conf reload.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 2, 2019, 1:35pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/2 "2019-04-02T13:35:50Z")

</div>

Working as expected. It reads the first line of the file. That does not match the pattern, so it adds it to an event (but does not flush the event). Then it reads the next line. That does match, so it flushes the previous event and adds line 2 to an event. It then reads every other line of the file and adds them to that event. It will not flush that event until it sees another line that matches root _or_ the pipeline is reloaded.

---

<div class="post-metadata">

**Author:** ![NativAt](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@NativAt](https://discuss.elastic.co/u/NativAt)\
**Post date:** [April 3, 2019, 4:33am UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/3 "2019-04-03T04:33:27Z")

</div>

So how do I solve it out? I do not want events which don't contain XPath, but when I'm using  
`pattern = "<\?xml version"` with `xpath => "/root/ChainId/text()", "ChainId"` I don't get anything.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 3, 2019, 1:51pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/4 "2019-04-03T13:51:44Z")

</div>

Does a file contain multiple root elements? If not, you can consume the entire file using a pattern that never matches and a timeout. For example

```
codec => multiline { pattern => "^Spalanzani" what => "previous" negate => true auto_flush_interval => 1 }
```

---

<div class="post-metadata">

**Author:** ![NativAt](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@NativAt](https://discuss.elastic.co/u/NativAt)\
**Post date:** [April 3, 2019, 2:05pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/5 "2019-04-03T14:05:45Z")

</div>

> [@Badger](#):
>
> auto\_flush\_interval =\> 1

Ok. I Change my pattern to suggested one, however now my XPath is not taking into effect. I get the following output:

```
logstash_1 | {
logstash_1 | "@timestamp" => 2019-04-03T14:03:04.092Z,
logstash_1 | "message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\n<root>\n <ChainId>7290027600007</ChainId>\n <SubChainId>001</SubChainId>\n <StoreId>001</StoreId>\n <BikoretNo>9</BikoretNo>\n <DllVerNo>8.0.1.3</DllVerNo>",
logstash_1 | "@version" => "1",
logstash_1 | "host" => "a4f1bf64a3d5",
logstash_1 | "tags" => [
logstash_1 | [0] "multiline"
logstash_1 | ],
logstash_1 | "type" => "xml",
logstash_1 | "path" => "/usr/share/logstash/logs/example8.xml"
logstash_1 | }
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 3, 2019, 2:13pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/6 "2019-04-03T14:13:19Z")

</div>

Your message does not contain \</root\>, which suggests your multiline codec is not what I wrote. No matter, even without closing the element the xpath expressions work for me. I don't know what else to suggest.

---

<div class="post-metadata">

**Author:** ![NativAt](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@NativAt](https://discuss.elastic.co/u/NativAt)\
**Post date:** [April 3, 2019, 2:19pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/7 "2019-04-03T14:19:50Z")

</div>

This is my exact conf file:

```
input {
  file {
    path => "/usr/share/logstash/logs/*.xml"
    type => "xml"
    # start_position => "beginning"
    sincedb_path => "/dev/null"
    codec => multiline {
      pattern => "^Spalanzani"
      negate => "true"
      what => "previous"
      auto_flush_interval => 1
    }
  }
}

filter {  
  xml {
    source => "message"
    store_xml => false
    # target => "root"
    # remove_namespaces => true
    xpath => [
        "/root/ChainId/text()", "ChainId",
        "/root/SubChainId/text()", "SubChainId",
        "/root/StoreId/text()", "StoreId",
        "/root/BikoretNo/text()", "BikoretNo",
        "/root/DllVerNo/text()", "DllVerNo"
    ]
  }
}

output {
  elasticsearch {
    hosts => "elasticsearch:9200"
    index => "xml_index"
  }

  stdout { 
    codec => rubydebug 
  }
}
```

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 3, 2019, 2:24pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/8 "2019-04-03T14:24:24Z")

</div>

It works for me.

```
   "message" => "<?xml version=\"1.0\" encoding=\"UTF-8\"?>\r\n<root>\r\n <ChainId>7290027600007</ChainId>\r\n <SubChainId>001</SubChainId>\r\n <StoreId>001</StoreId>\r\n <BikoretNo>9</BikoretNo>\r\n <DllVerNo>8.0.1.3</DllVerNo>\r\n</root>\r",
   "ChainId" => [
    [0] "7290027600007"
],

```

etc.

---

<div class="post-metadata">

**Author:** ![NativAt](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@NativAt](https://discuss.elastic.co/u/NativAt)\
**Post date:** [April 3, 2019, 2:36pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/9 "2019-04-03T14:36:10Z")

</div>

It's really strange! I don't understand what may be the cause for this.

Are you running on docker?  
Which Logstash version are you running?  
Does the file is crlf or lf?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [April 3, 2019, 2:37pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/10 "2019-04-03T14:37:57Z")

</div>

I am running 6.6.0 on Linux. Not docker.

The newlines have to be native to whatever platform you are running on. If you have Windows newlines on a UNIX platform you need to mutate+gsub the \r's out of the message.

---

<div class="post-metadata">

**Author:** ![NativAt](https://avatars.discourse-cdn.com/v4/letter/n/fbc32d/32.png) [@NativAt](https://discuss.elastic.co/u/NativAt)\
**Post date:** [April 3, 2019, 3:14pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/11 "2019-04-03T15:14:10Z")

</div>

I found the problem! My XML files were encoded with  
`UTF-8 BOM` instead of `UTF-8`.

_ **The Solution Lies Within the Problem** _  
[![](https://media.giphy.com/media/1LixmQhqqw9SE/giphy.gif) ](https://media.giphy.com/media/1LixmQhqqw9SE/giphy.gif)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 1, 2019, 3:14pm UTC](https://discuss.elastic.co/t/logstash-xml-file-parsing-awkward-splitting-event-problem/174906/12 "2019-05-01T15:14:18Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.