# Logstash hanging while split array (consume too much memory)

**URL:** <https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726>\
**Category:** Logstash\
**Created:** [June 27, 2019, 6:05am UTC](https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726 "2019-06-27T06:05:35Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![taewoo\_Roh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taewoo_roh/32/48887_2.png) [@taewoo\_Roh](https://discuss.elastic.co/u/taewoo_Roh)\
**Post date:** [June 27, 2019, 6:05am UTC](https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726/1 "2019-06-27T06:05:35Z")

</div>

Hello all.

I have a problem while using logstash split filter.

Logstash extract data from API using http\_polled input plug-in.  
and returned data contain array (format is json)  
I split array data using split filter plug-in.

There is no problem when array contain small a mount of data. (about 50~60)

But there is problem when array contain large numbers of data (about 5000)

Logstash consume too much memory while split array.  
And jvm hanging when logstash max memory of jvm.

I have increase jvm memory up to 128GB but still logstash need more memory.

Logstash looks like store spit data on memory before finish split filter.

Is there any option I can send event to output (elasticsearch) while split filter working?

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [June 27, 2019, 2:07pm UTC](https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726/2 "2019-06-27T14:07:10Z")

</div>

Does [this](https://discuss.elastic.co/t/how-to-split-into-multiple-events-dynamically-for-a-given-json-tried-from-various-question-in-forums/167745/10) help? If you are splitting on a very large field then the split filter clones the target field before overwriting it. Obviously that can be optimized to remove the target before cloning it.

---

<div class="post-metadata">

**Author:** ![taewoo\_Roh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/taewoo_roh/32/48887_2.png) [@taewoo\_Roh](https://discuss.elastic.co/u/taewoo_Roh)\
**Post date:** [June 28, 2019, 4:43am UTC](https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726/3 "2019-06-28T04:43:43Z")

</div>

Thank you.

problem solved.

I changed 'split.rb' source by your recommend.  
And threre is more high memory consume and performance issue.

Thank you again.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 26, 2019, 4:43am UTC](https://discuss.elastic.co/t/logstash-hanging-while-split-array-consume-too-much-memory/187726/4 "2019-07-26T04:43:44Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.