# Large XML crashes logstash with OOM

**URL:** <https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682>\
**Category:** Logstash\
**Created:** [September 4, 2015, 12:19pm UTC](https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682 "2015-09-04T12:19:57Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![asatsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asatsi/32/28417_2.png) [@asatsi](https://discuss.elastic.co/u/asatsi)\
**Post date:** [September 4, 2015, 12:19pm UTC](https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682/1 "2015-09-04T12:19:57Z")

</div>

Hi,

Working on a setup with large XML files as part of the logs. The XMLs at times could be 10,20MB large. Facing issues with such large files and logstash crashing with OutOfMemory error, or entity expansion grown too large errors intermittently. Anyone came across such situation and know a way out? Would really appreciate your help in this regard.

Cheers,  
Satish/

---

<div class="post-metadata">

**Author:** ![asatsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asatsi/32/28417_2.png) [@asatsi](https://discuss.elastic.co/u/asatsi)\
**Post date:** [September 5, 2015, 4:46am UTC](https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682/2 "2015-09-05T04:46:11Z")

</div>

@PhaedrusTheGreek It could be reproduced using the below configuration file:

```
input
{
        file
        {
                path => "/tmp/xml.log"
        }
}

filter
{
        multiline
        {
                pattern => "^<ns0:"
                negate => true
                what => previous
        }
        xml
        {
                source => ["message"]
                target => ["x"]
        }
}

output
{
        stdout
        {
                codec => rubydebug
        }
}

```

The input XML gist is here: [https://gist.github.com/asatsi/330e5c23830752d53bee](https://gist.github.com/asatsi/330e5c23830752d53bee)

FYI, I am running logstash 1.5.3.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [September 6, 2015, 2:20pm UTC](https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682/3 "2015-09-06T14:20:53Z")

</div>

Logstash's default JVM heap is 500 MB and I think that should be enough for parsing a 20 MB XML file. Have you tried increasing the heap size? Depending on how you start Logstash you can do that via /etc/default/logstash, /etc/sysconfig/logstash, or by setting the LS\_HEAP\_SIZE environment variable. Try "1024m" for starters.

---

<div class="post-metadata">

**Author:** ![asatsi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/asatsi/32/28417_2.png) [@asatsi](https://discuss.elastic.co/u/asatsi)\
**Post date:** [September 8, 2015, 10:13am UTC](https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682/4 "2015-09-08T10:13:39Z")

</div>

Increasing the heap size to 4096m helped avoid the crashes for now. Thanks!

---

<div class="post-metadata">

**Author:** ![PhaedrusTheGreek](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/phaedrusthegreek/32/4884_2.png) [@PhaedrusTheGreek](https://discuss.elastic.co/u/PhaedrusTheGreek)\
**Post date:** [September 8, 2015, 4:11pm UTC](https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682/5 "2015-09-08T16:11:54Z")

</div>

@asatsi, I am able to reproduce the problem using your configuration and the provided XML file. Because each XML object is 20MB, this is expected. The XML DOM parsing library explodes each XML document into a much larger object in memory, so the only workaround would be to do as you did and increase the Java Heap size.

Also, please use caution if you intend to aggressively index documents of these size into Elasticsearch. Search and Aggregation should perform well, but Indexing, Retrieving and Merging will be Disk intensive.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:29am UTC](https://discuss.elastic.co/t/large-xml-crashes-logstash-with-oom/28682/6 "2017-07-06T05:29:44Z")

</div>