Logstash hanging while split array (consume too much memory)

Hello all.

I have a problem while using logstash split filter.

Logstash extract data from API using http_polled input plug-in.
and returned data contain array (format is json)
I split array data using split filter plug-in.

There is no problem when array contain small a mount of data. (about 50~60)

But there is problem when array contain large numbers of data (about 5000)

Logstash consume too much memory while split array.
And jvm hanging when logstash max memory of jvm.

I have increase jvm memory up to 128GB but still logstash need more memory.

Logstash looks like store spit data on memory before finish split filter.

Is there any option I can send event to output (elasticsearch) while split filter working?

Does this help? If you are splitting on a very large field then the split filter clones the target field before overwriting it. Obviously that can be optimized to remove the target before cloning it.

1 Like

Thank you.

problem solved.

I changed 'split.rb' source by your recommend.
And threre is more high memory consume and performance issue.

Thank you again.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.