# Logstash going OOM while dumping data from ES to CSV

**URL:** <https://discuss.elastic.co/t/logstash-going-oom-while-dumping-data-from-es-to-csv/33491>\
**Category:** Logstash\
**Created:** [November 2, 2015, 8:43am UTC](https://discuss.elastic.co/t/logstash-going-oom-while-dumping-data-from-es-to-csv/33491 "2015-11-02T08:43:47Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![ankmathur14](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@ankmathur14](https://discuss.elastic.co/u/ankmathur14)\
**Post date:** [November 2, 2015, 8:43am UTC](https://discuss.elastic.co/t/logstash-going-oom-while-dumping-data-from-es-to-csv/33491/1 "2015-11-02T08:43:47Z")

</div>

Hello all,

I have 60 timestamp wise indexes in my cluster (total containing 200 million documents (150 GB) )  
I am trying to export some data (around 15 million) from these indexes to a csv file using logstash.  
Logstash is going OOM when i try to dump all indexes at the same time, although it is working fine with one index at a time.

bin/logstash.bat agent -f myconfig.config

io/console not supported; tty will not be manipulated  
Logstash startup completed  
java.lang.OutOfMemoryError: Java heap space  
Dumping heap to java\_pid1900.hprof ...  
Heap dump file created [359502197 bytes in 3.316 secs]  
Exception in thread "\<elasticsearch" java.lang.UnsupportedOperationException  
at java.lang.Thread.stop(Thread.java:869)  
at org.jruby.RubyThread.exceptionRaised(RubyThread.java:1221)  
at org.jruby.internal.runtime.RubyRunnable.run(RubyRunnable.java:112)  
at java.lang.Thread.run(Thread.java:745)  
Logstash shutdown completed

config :

input {  
elasticsearch {  
}  
}

output {  
csv {  
}  
}

Logstash 1.5  
Elasticsearch 1.6  
Total memory of cluster : 22 GB

What do i have to do to resolve this?  
OR any other way to export ES data to csv/text file?

Thanks.  
Ankur

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [November 2, 2015, 9:21am UTC](https://discuss.elastic.co/t/logstash-going-oom-while-dumping-data-from-es-to-csv/33491/2 "2015-11-02T09:21:20Z")

</div>

Based on that config you are taking _everything_ from ES and exporting it.

You may want to either increase the heap you're giving to LS, or reduce the default [size](https://www.elastic.co/guide/en/logstash/current/plugins-inputs-elasticsearch.html#plugins-inputs-elasticsearch-size).

---

<div class="post-metadata">

**Author:** ![ankmathur14](https://avatars.discourse-cdn.com/v4/letter/a/7feea3/32.png) [@ankmathur14](https://discuss.elastic.co/u/ankmathur14)\
**Post date:** [November 2, 2015, 9:49am UTC](https://discuss.elastic.co/t/logstash-going-oom-while-dumping-data-from-es-to-csv/33491/3 "2015-11-02T09:49:03Z")

</div>

Thanks for your reply.  
Here is my actual config :

input {  
elasticsearch {  
hosts =\> "10.10.8.14"  
query =\> '{ "query": { "match": { "FileType": "TS\_FILE" } } }'  
index =\> "logstash-srsdb-\*"  
size =\> 10  
scroll =\> "30s"

}  
}

output {  
#stdout {}  
csv {  
fields =\> ["Col1", "Col5", "Col7"]  
path =\> "D:\elastic\_dump\dump.csv"  
csv\_options =\> {"col\_sep" =\> "," "row\_sep" =\> "\r\n"}  
}  
}

I have reduced the size from default to 10. Now it is NOT going OOM but taking long time to dump.

How to increase heap size of Logstash? I tried to set LS\_HEAP\_SIZE in env variable but no luck.

One more thing using this line "csv\_options =\> {"col\_sep" =\> "," "row\_sep" =\> "\r\n"}", row\_sep actually printing "\r\n" in text instead of new line.

Thanks.  
Ankur

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 5:24am UTC](https://discuss.elastic.co/t/logstash-going-oom-while-dumping-data-from-es-to-csv/33491/4 "2017-07-06T05:24:33Z")

</div>