Logstash hanging while split array (consume too much memory)

Elastic Stack Logstash
1

Hello all.

I have a problem while using logstash split filter.

Logstash extract data from API using http_polled input plug-in.
and returned data contain array (format is json)
I split array data using split filter plug-in.

There is no problem when array contain small a mount of data. (about 50~60)

But there is problem when array contain large numbers of data (about 5000)

Logstash consume too much memory while split array.
And jvm hanging when logstash max memory of jvm.

I have increase jvm memory up to 128GB but still logstash need more memory.

Logstash looks like store spit data on memory before finish split filter.

Is there any option I can send event to output (elasticsearch) while split filter working?

2

Does this help? If you are splitting on a very large field then the split filter clones the target field before overwriting it. Obviously that can be optimized to remove the target before cloning it.

3

Thank you.

problem solved.

I changed 'split.rb' source by your recommend.
And threre is more high memory consume and performance issue.

Thank you again.