# Read huge elastic index with logstash

**URL:** <https://discuss.elastic.co/t/read-huge-elastic-index-with-logstash/360646>\
**Category:** Logstash\
**Created:** [May 31, 2024, 6:48pm UTC](https://discuss.elastic.co/t/read-huge-elastic-index-with-logstash/360646 "2024-05-31T18:48:05Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![gueri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gueri/32/103243_2.png) [@gueri](https://discuss.elastic.co/u/gueri)\
**Post date:** [May 31, 2024, 6:48pm UTC](https://discuss.elastic.co/t/read-huge-elastic-index-with-logstash/360646/1 "2024-05-31T18:48:05Z")

</div>

Hi,

I'm running Logstash 8.7 and reading from Elasticsearch 8.13.  
I want to retrieve all docs from an index with 18 millions logs and copy to Kafka.  
I thought Logstash was capable of doing that with the elasticsearch input plugin and scroll pagination but I can't.  
I also try different values for size or scoll without success.

> java.lang.OutOfMemoryError: Java heap space  
> Dumping heap to java\_pid3331477.hprof ...  
> [81.561s][error][jvmti] Posting Resource Exhausted event: Java heap space

```auto
input {
   elasticsearch {
      hosts => "my_host:9200"
      user => "logstash"
      password => "xxxxxxxxxxxxx"
      ssl => true
      ca_file => "ca.pem"
      index => "my_big_index"
      #size => ...
      #scroll => ...
   }
}

filter {
}

output {
   kafka {
      codec => json
      bootstrap_servers => "my_server:9092"
      topic_id => "my_topic"
      security_protocol => "SSL"
      ssl_keystore_location => "certstore.jks"
      ssl_keystore_password => "xxxxxxxxxxxxxx"
      ssl_keystore_type => "jks"
      ssl_truststore_location => "certstore.jks"
      ssl_truststore_password => "xxxxxxxxxxxxxx"
      ssl_truststore_type => "jks"
   }
}

```

if I test with a small index, it works fine. But with my big index, no way!

Ok I have a message with OutOfMemoryError but the scroll pagination should allow reading to be delayed no ?

Have you any idea to help me ?

Should I ask our system operator to change some parameters on logstash configuration ?

Regards

---

<div class="post-metadata">

**Author:** ![gueri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/gueri/32/103243_2.png) [@gueri](https://discuss.elastic.co/u/gueri)\
**Post date:** [July 10, 2024, 4:04pm UTC](https://discuss.elastic.co/t/read-huge-elastic-index-with-logstash/360646/2 "2024-07-10T16:04:58Z")

</div>

Hi,

this was a malfunction from our logstash, even after stop/restart  
Now with logstash 8.9 this problem no longer exists

Regards,