# java.lang.OutOfMemoryError: Java heap space while indexing xml file

**URL:** <https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082>\
**Category:** Logstash\
**Created:** [August 7, 2017, 10:21am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082 "2017-08-07T10:21:39Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![aashishchauhan06](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@aashishchauhan06](https://discuss.elastic.co/u/aashishchauhan06)\
**Post date:** [August 7, 2017, 10:21am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082/1 "2017-08-07T10:21:39Z")

</div>

HI,

I'm indexing a XML file and given ElasticSearch as output but after approx. 2 hours i got OOM error like this

```
java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid6940.hprof ...
Heap dump file created [9277681492 bytes in 117.530 secs]
[2017-08-04T23:02:13,255][ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. {"exception"=>java.lang.OutOfMemoryError: Java heap space, "backtrace"=>[]}
[2017-08-04T23:02:15,181][ERROR][logstash.pipeline] Exception in pipelineworker, the pipeline stopped processing new events, please check your filter configuration and restart Logstash. 
Error: Your application used more memory than the safety cap of 6G.
Specify -J-Xmx####m to increase it (#### = cap size in MB).
Specify -w for full OutOfMemoryError stack trace

```

Here is my configuration file:

```
input 
{
	file {
        path => "path/drug.xml"
		type => "drugbank"
		start_position => beginning
		sincedb_path => "/dev/null"
		codec => multiline
				{
					pattern => "^<\?drugbank.*\>"
					negate => true
					what => "previous"
					max_bytes => "400 MiB"
					max_lines => "5000000000"

				}
		}
}

filter {
        xml {
                source => "message"
target => "xmldata"
store_xml => "false"
        xpath => ["/drugbank/drug", "drug"]
       
 }
       
 mutate {
            remove_field => ["message", "inxml", "xmldata"]
        }

        split {
                field => "[drug]"
        }

        xml {
                source => "drug"
                store_xml => "false"
				xpath => ["/drug/drugbank-id/text()", "Drug ID"]
				xpath => ["/drug/name/text()", "Drug name"]
				xpath => ["/drug/targets/target/polypeptide/gene-name/text()", "Gene"]
        }
		 mutate {
                replace => {
				"Drug ID" => "%{[Drug ID][0]}"
                "Drug name" => "%{[Drug name][0]}"
                "Gene" => "%{[Gene][0]}"
                }
			}	
    mutate {
        remove_field => ["drug"]
    }
    }
    output
    { 
     elasticsearch {
       codec => json
       hosts => ["10.xx.xx.xx2","10.xx.xx.xx4"]
       index => "test_index"
     }
    }

```

Any suggestion much appreciated.

Thanks

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [August 7, 2017, 11:25am UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082/2 "2017-08-07T11:25:26Z")

</div>

How big is the file? Have you verified that the multiline codec is doing the right thing?

---

<div class="post-metadata">

**Author:** ![aashishchauhan06](https://avatars.discourse-cdn.com/v4/letter/a/bc8723/32.png) [@aashishchauhan06](https://discuss.elastic.co/u/aashishchauhan06)\
**Post date:** [August 7, 2017, 12:49pm UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082/3 "2017-08-07T12:49:12Z")

</div>

File size is 397 MB and it contains approx 10 million lines. Yes i have verified multiline codec is working fine with a small sample file with 7-8 records of 5 thousand lines.

Initially what happening is, records got truncated because of default setting which is 500 lines MAX\_LINE and 10 MB MAX\_BYTE so i changed it to 5000000000 and 400 MB respectively afterwards my code start working fine.

I have ELK cluster on 3 machines (Coordinator, Master, DataNode ) and also set LS\_HEAP\_SIZE=12g and ES\_HEAP\_SIZE=12g and JVM max size is 6g.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 4, 2017, 12:49pm UTC](https://discuss.elastic.co/t/java-lang-outofmemoryerror-java-heap-space-while-indexing-xml-file/96082/4 "2017-09-04T12:49:27Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.