Read huge elastic index with logstash

Elastic Stack Logstash
1

Hi,

I'm running Logstash 8.7 and reading from Elasticsearch 8.13.
I want to retrieve all docs from an index with 18 millions logs and copy to Kafka.
I thought Logstash was capable of doing that with the elasticsearch input plugin and scroll pagination but I can't.
I also try different values for size or scoll without success.

java.lang.OutOfMemoryError: Java heap space
Dumping heap to java_pid3331477.hprof ...
[81.561s][error][jvmti] Posting Resource Exhausted event: Java heap space

input {
   elasticsearch {
      hosts => "my_host:9200"
      user => "logstash"
      password => "xxxxxxxxxxxxx"
      ssl => true
      ca_file => "ca.pem"
      index => "my_big_index"
      #size => ...
      #scroll => ...
   }
}

filter {
}

output {
   kafka {
      codec => json
      bootstrap_servers => "my_server:9092"
      topic_id => "my_topic"
      security_protocol => "SSL"
      ssl_keystore_location => "certstore.jks"
      ssl_keystore_password => "xxxxxxxxxxxxxx"
      ssl_keystore_type => "jks"
      ssl_truststore_location => "certstore.jks"
      ssl_truststore_password => "xxxxxxxxxxxxxx"
      ssl_truststore_type => "jks"
   }
}

if I test with a small index, it works fine. But with my big index, no way!

Ok I have a message with OutOfMemoryError but the scroll pagination should allow reading to be delayed no ?

Have you any idea to help me ?

Should I ask our system operator to change some parameters on logstash configuration ?

Regards

2

Hi,

this was a malfunction from our logstash, even after stop/restart
Now with logstash 8.9 this problem no longer exists

Regards,

1 Like