Logstash java heap memory causing delay to pushing events to elasticsearch

Elastic Stack Logstash
1

Recently we made changes to java heap memory from 1G to 4G. Sometime it was stopped as it was not possible to process the event.

when we increased the java heap memory logstash is processing the log events but elasticsearch is having issue with few index where we have high log events to be ingested.

  1. JVM memory
  2. CPU
  3. Storage

seems to be within the limit.

Please advise

2

Hello,

You need to provide more context.

Are Logstash and Elasticsearch on the same machine? What are the specs?

3

Logstash runs as a service in ec2 to push logs.

Elasticsearch runs on VM on AWS (8 cpu and 64G of memory) 16 nodes. Things was working fine until we increased the java heap memory.

4

And what are the specs for the Logstash host? You didn't provide this information.

If you increased the heap from 1G to 4G, you should have increased the machine type as well to at least 8 GB.

5

Logstash runs on VM where the we are using 16G of memory and 4 vcpu. The problem is not with logstash. Logstash does the ingestion correctly. But ES is not able to accommodate.

The change to heap memory was done as stated in documentation not more than 4G.

6

So is your issue is with Elasticsearch and not Logstash?

Please provide more context what exactly was changed because it is a little confusing what is really your issue.

After you changed the heap for Logstash, your Elasticsearch cluster cannot keep-up with the events anymore?

Do you have any logs from Elasticsearch? What is the configuration for your Elasticsearch machines, the Heap configured and specially the disk type (hdd, sdd, nvme).

7

I have changed the java heap memory for logstash. Logstash is the agent on the windows VM on AWS 8 core cpu and 32 gigs of memory. Application as well runs on this. We have a pipeline we logstash take the input and send it to elastic.

previouslt logstash was getting stopped in production environments saying java heap memory was low.

as stated in the documentation we changed it to min and max to 4G.

elasticsearch runs on 8 hot nodes and 2 warm nodes. 64 gigs of RAM and 16 core CPU. EBS volume of gp3 1000 iops and 500 throughput.

Everything was fine means ec2 windows instance logstash used to stopped sometime while the memory was not enough. But now we increased.

Elasticsearch is not able to handle the load.

errors reported on logstash-plain.log

[2025-01-28T12:01:07,362][ERROR][logstash.outputs.elasticsearch][main][82dc87e46666b2a4ed3d4390656dccdb4da21d1e457dce4fc50bae6d03] Encountered a retryable error (will retry with exponential backoff) {:code=>503, :url=>"https://******************************************************:443/_bulk", :content_length=>752}

the log events in kibana is having 4 days lag. this is with windows VM only. Linux VM runs fine.