# How to quikly parsing 1 million xml

**URL:** <https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141>\
**Category:** Logstash\
**Created:** [January 29, 2017, 9:24pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141 "2017-01-29T21:24:42Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [January 29, 2017, 9:24pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/1 "2017-01-29T21:24:42Z")

</div>

Hello.  
I have 1 million file with xml. Each file size from a few kilobytes to 20mb.  
I need parsing their and put to elastic. I tried to use the next config:

> file {  
> path =\> "/opt/lun1/data-unzip/ftp/228-pur-plan/_.xml_"  
> exclude =\> "\*.zip"  
> type =\> "228-pur-plan"  
> start\_position =\> beginning  
> max\_open\_files =\> "64"  
> close\_older =\> "0.1"  
> codec =\> multiline {  
> pattern =\> "xml version"  
> negate =\> true  
> what =\> "previous"  
> max\_lines =\> "999999"  
> max\_bytes =\> "20 MiB"  
> }  
> }

> filter {  
> if [type] =~ "228-pur-plan" {  
> xml {  
> store\_xml =\> false  
> source =\> "message"  
> xpath =\> [  
> \<about\_100\_lines\>  
> ]  
> }

> output {  
> if [type] =~ "228-pur-plan" {  
> elasticsearch {  
> hosts =\> "127.0.0.1:9200"  
> index =\> "228-pur-plan-%{+YYYY.MM}"  
> template =\> "/opt/logstash/custom\_patterns/elasticsearch-template\_228.json"  
> template\_name =\> "228"  
> }  
> }  
> }

My hardware server use CentOS 6.8 and have:  
80Gb memory  
Intel(R) Xeon(R) CPU E5620 @ 2.40GHz  
with 16 cpus.

I set logstash heap size 32gb.

But it work so slowly! I have very bad result - 4 file parses per second.  
Please, how I do more quickly ? May be I must do any tuning ? OR may be I should use other input plugin?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [January 30, 2017, 5:03am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/2 "2017-01-30T05:03:23Z")

</div>

Try increasing the pipeline worker count.  
Also, is it LS or ES that is the bottleneck?

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [January 30, 2017, 5:55am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/3 "2017-01-30T05:55:56Z")

</div>

> [@warkolm](#):
>
> Also, is it LS or ES that is the bottleneck?

How can I check it?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 30, 2017, 6:32am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/4 "2017-01-30T06:32:12Z")

</div>

> How can I check it?

What's the limiting factor? How many files per second can Logstash process alone? How many documents per second can Elasticsearch accept? Measure.

I think a major issue is that you're using a single file input which is going to run in a single thread. Splitting it into multiple file inputs is likely to help, but if you really want to improve the performance I suspect writing a program that bulk-reads XML files and feeds them to Logstash via e.g. TCP would perform much better.

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [January 30, 2017, 7:03am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/5 "2017-01-30T07:03:14Z")

</div>

> [@magnusbaeck](#):
>
> TCP would perform much better.

ОК. I tried tcp input too, but I have error

> [FATAL][logstash.runner] An unexpected error occurred! {:error=\>#\<Nokogiri::XML::XPath::SyntaxError: /ns2:purchasePlan/_[local-name()='header']/_[local-name()='guid']/text()\>, :backtrace=\>["nokogiri/XmlXpathContext.java:169:in `evaluate'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/nokogiri-1.6.8.1-java/lib/nokogiri/xml/searchable.rb:165:in `xpath'", "org/jruby/RubyArray.java:2414:in `map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/nokogiri-1.6.8.1-java/lib/nokogiri/xml/searchable.rb:156:in `xpath'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-4.0.2/lib/logstash/filters/xml.rb:153:in `filter'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-4.0.2/lib/logstash/filters/xml.rb:152:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do\_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `multi_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter_delegator.rb:41:in `multi\_filter'", "(eval):155:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):152:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):127:in `filter_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:295:in `filter\_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:192:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped_synchronous_queue.rb:191:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:294:in `filter_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:282:in `worker\_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:258:in `start\_workers'"]}

My tcp config;

> tcp {  
> host =\> "0.0.0.0"  
> port =\> 7101  
> type =\> "228-pur-plan"  
> codec =\> multiline {  
> pattern =\> "xml version"  
> negate =\> true  
> what =\> "previous"  
> max\_lines =\> "999999"  
> max\_bytes =\> "20 MiB"  
> }

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 30, 2017, 7:05am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/6 "2017-01-30T07:05:25Z")

</div>

Always post logs as preformatted text. As it stands the interesting error message has been mangled and isn't visible.

Don't use the multiline codec. Whatever feeds Logstash via TCP should slurp the whole file in one go, removing newline characters if necessary.

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [January 30, 2017, 8:47am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/7 "2017-01-30T08:47:07Z")

</div>

> [@magnusbaeck](#):
>
> Don't use the multiline codec. Whatever feeds Logstash via TCP should slurp the whole file in one go, removing newline characters if necessary.

I try to use without multiline codec:

> tcp {  
> host =\> "0.0.0.0"  
> port =\> "7101"  
> type =\> "28-pur-plan"  
> }

> filter {  
> if [type] =~ "228-pur-plan" {  
> xml {  
> store\_xml =\> false  
> source =\> "message"  
> xpath =\> [  
> about 100 lines  
> ]  
> }

> output {  
> if [type] =~ "228-pur-plan" {  
> elasticsearch {  
> hosts =\> "127.0.0.1:9200"  
> index =\> "228-pur-plan-%{+YYYY.MM}"  
> template =\> "/opt/logstash/custom\_patterns/elasticsearch-template\_228.json"  
> template\_name =\> "228"  
> }  
> }  
> }

And I have the next error messages:

`[2017-01-30T11:41:03,942][ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>#<Nokogiri::XML::XPath::SyntaxError: /ns2:purchasePlan/*[local-name()='header']/*[local-name()='guid']/text()>, "backtrace"=>["nokogiri/XmlXpathContext.java:169:in `evaluate'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/nokogiri-1.6.8.1-java/lib/nokogiri/xml/searchable.rb:165:in `xpath'", "org/jruby/RubyArray.java:2414:in `map'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/nokogiri-1.6.8.1-java/lib/nokogiri/xml/searchable.rb:156:in `xpath'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-4.0.2/lib/logstash/filters/xml.rb:153:in `filter'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/vendor/bundle/jruby/1.9/gems/logstash-filter-xml-4.0.2/lib/logstash/filters/xml.rb:152:in `filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:145:in `do_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:164:in `multi\_filter'", "org/jruby/RubyArray.java:1613:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/filters/base.rb:161:in `multi\_filter'", "/usr/share/logstash/logstash-core/lib/logstash/filter\_delegator.rb:41:in `multi_filter'", "(eval):155:in `initialize'", "org/jruby/RubyArray.java:1613:in `each'", "(eval):152:in `initialize'", "org/jruby/RubyProc.java:281:in `call'", "(eval):127:in `filter\_func'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:295:in `filter_batch'", "org/jruby/RubyProc.java:281:in `call'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:192:in `each'", "org/jruby/RubyHash.java:1342:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/util/wrapped\_synchronous\_queue.rb:191:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:294:in `filter\_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:282:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:258:in `start\_workers'"]}`

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [January 30, 2017, 8:55am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/8 "2017-01-30T08:55:19Z")

</div>

Okay, so the xml filter thinks it gets garbage input. I suggest you remove newline characters from the input so that what you send to Logstash contains one file per line, but that's just a guess at what's wrong. Look at the event the xml filter is complaining about.

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [January 30, 2017, 1:16pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/9 "2017-01-30T13:16:02Z")

</div>

I do it. I remove newline characters and forward to logstash throw TCP for 3 inputs

> tcp {  
> host =\> "0.0.0.0"  
> port =\> "7101"  
> type =\> "28-pur-plan"  
> }  
> tcp {  
> host =\> "0.0.0.0"  
> port =\> "7102"  
> type =\> "28-pur-plan"  
> }  
> tcp {  
> host =\> "0.0.0.0"  
> port =\> "7103"  
> type =\> "28-pur-plan"  
> }

But I have bad result: 4-10 xml per second  
I tried output to file (instead elasticsearch) but I have 4-10 xml per second too.

Where is my bottelneck ? I think this is logstash, but where?

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [January 30, 2017, 2:09pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/10 "2017-01-30T14:09:09Z")

</div>

I tried to remove xpath xml parsing, and i have good result - over 100 xml rep sec.  
Parsing is a bottleneck. How can I optimize xpath parsing or increase the amount of resources allocated to parse?

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [January 30, 2017, 3:28pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/11 "2017-01-30T15:28:29Z")

</div>

The XML support in Logstash is done by an external library which is fine for small use cases but it is **very slow** for xpath and other complex use cases.

You could consider an XML/XSLT to JSON preprocessor before Logstash. Example: [https://github.com/bramstein/xsltjson](https://github.com/bramstein/xsltjson)

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [February 1, 2017, 6:42am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/12 "2017-02-01T06:42:57Z")

</div>

OK. I convert my xml to json and test logstash parsing. My config:

```
input{
        unix {
                path => "/tmp/socket1"
                mode => "server"
                type => "228-pur-plan-soc"
                codec => "json_lines"
        }
        unix {
                path => "/tmp/socket2"
                mode => "server"
                type => "228-pur-plan-soc"
                codec => "json_lines"
        }
        unix {
                path => "/tmp/socket3"
                mode => "server"
                type => "228-pur-plan-soc"
                codec => "json_lines"
        }
        tcp {
                host => "0.0.0.0"
                port => "7101"
                type => "228-pur-plan-tcp"
                codec => "json_lines"
        }
        tcp {
                host => "0.0.0.0"
                port => "7102"
                type => "228-pur-plan-tcp"
                codec => "json_lines"
        }
        tcp {
                host => "0.0.0.0"
                port => "7103"
                type => "228-pur-plan-tcp"
                codec => "json_lines"
        }
}

filter {
   if [type] =~ "228-pur-plan-soc" {
        json { source => "message" }
        metrics {
                meter => "soc_events"
                add_tag => "metrics"
        }
   }

   if [type] =~ "228-pur-plan-tcp" {
        json { source => "message" }
        metrics {
                meter => "tcp_events"
                add_tag => "metrics"
        }
   }

}

output {
         if [type] =~ "228-pur-plan" {
                file {
                    path => "/opt/lun1/out.dat"
                }
       }
        if "metrics" in [tags] {
                elasticsearch {
                        hosts => "127.0.0.1:9200"
                        index => "ls_metrics-%{+YYYY.MM}"
                }
        }
}

```

And I again got bad results. How can I take more speed?

 ![](https://us1.discourse-cdn.com/elastic/original/2X/1/1009c626800c294e00587edbb88f906cb0bef7ff.JPG)

---

<div class="post-metadata">

**Author:** ![Hi\_pay](https://avatars.discourse-cdn.com/v4/letter/h/858c86/32.png) [@Hi\_pay](https://discuss.elastic.co/u/Hi_pay)\
**Post date:** [February 1, 2017, 10:14am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/13 "2017-02-01T10:14:24Z")

</div>

Beats пробовал настроить?  
Он предназначен как раз для чтения файлов.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [February 1, 2017, 10:15am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/14 "2017-02-01T10:15:48Z")

</div>

English please.

---

<div class="post-metadata">

**Author:** ![Hi\_pay](https://avatars.discourse-cdn.com/v4/letter/h/858c86/32.png) [@Hi\_pay](https://discuss.elastic.co/u/Hi_pay)\
**Post date:** [February 1, 2017, 10:23am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/15 "2017-02-01T10:23:16Z")

</div>

You tried to set up the Beats (Filebeat)?  
It is designed just for reading files and sending in ES

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [February 1, 2017, 11:45am UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/16 "2017-02-01T11:45:57Z")

</div>

Do you mean that my problem with input?

---

<div class="post-metadata">

**Author:** ![Hi\_pay](https://avatars.discourse-cdn.com/v4/letter/h/858c86/32.png) [@Hi\_pay](https://discuss.elastic.co/u/Hi_pay)\
**Post date:** [February 1, 2017, 12:20pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/17 "2017-02-01T12:20:09Z")

</div>

What speed do you expect and what is the speed of your hard drive?  
ES is on the same server where the files are read?

4-10 xml per second ~ 80-200 Mb/s - average hdd speed  
perhaps you used ssd?

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [February 1, 2017, 12:29pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/18 "2017-02-01T12:29:23Z")

</div>

I expect more than 50 json (or xml) per sec.

I tried test in another server (Intel(R) Xeon(R) CPU E5-2680 v2 @ 2.80GHz with 40 CPUs), and I have the same result - 4-10 json per sec.

---

<div class="post-metadata">

**Author:** ![guyboertje](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/guyboertje/32/31592_2.png) [@guyboertje](https://discuss.elastic.co/u/guyboertje)\
**Post date:** [February 2, 2017, 5:48pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/19 "2017-02-02T17:48:48Z")

</div>

If you want to test whether ES is your bottleneck you could try the redis-output talking to Redis. This will use bulk updates and networking and therefore approximates the ES output and downstream.

---

<div class="post-metadata">

**Author:** ![111126](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/111126/32/36818_2.png) [@111126](https://discuss.elastic.co/u/111126)\
**Post date:** [February 2, 2017, 8:56pm UTC](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141/20 "2017-02-02T20:56:45Z")

</div>

> [@guyboertje](#):
>
> If you want to test whether ES is your bottleneck you could try the redis-output talking to Redis

Ok. I tested it. My results, you can view at [Failed to send event to Redis](https://discuss.elastic.co/t/failed-to-send-event-to-redis/73581)  
🙂

[Next page](https://discuss.elastic.co/t/how-to-quikly-parsing-1-million-xml/73141.md?page=2)