How to avoid jvm error in python elasticsearch rest client

Elastic Stack Elasticsearch
1

I have 10 elasticsearch nodes in one server(cluster), all nodes have 2gb jvm heap size.

I have a python rest client for elasticsearch. With that client I fetch datas from elasticsearch and re-indexing with new datas.

I don't want rest client use full of jvm. Cause, after some times elasticsearch nodes shutdown because of jvm heap size error. When jvm heap size is near to full, I want that client make empty the garbage collector. How can i do this in Python ?

Thanks for answering

2

Why would you have 10 nodes with such small heap on a single host? You need to have fewer nodes with larger heap for your use case. You can not manually trigger garbage collection from external processes.

3

I have 10 nodes and each node has 2gb jvm, so elasticsearch use 20gb of ram. Is it small heap? I didn't understand what is the point of removing some nodes and increasing other nodes heap size? and what about the datas if i do this, I wouldn't able to reach all data ? Is it possible to recover data when decreasing the node counts?

4

I do not see the point of running so many nodes with a 2GB heap (this is small) on a single host. There is some overhead per node in terms of cluster state and mappings, so having 10 copies of this would use up more heap compared to a single node with the same total amount of heap. I single node with 20GB of heap would probably also perform better as there would be less communication and coordination overhead. Why did you set it up like this in the first place? What were you hoping to gain?

1 Like
5

I have chosen this because of red state. With one node, I get red state. Because there is no other nodes to store replicas. I have 6 eligible master nodes, and others are configured.
If I chosen to move 10 nodes to one node? what about my data? Am i able to recover ?

6

Red means you have unallocated primaries.
Yellow means you have unallocated replicas.
If you had one node, you would more than likely see yellow. If you are seeing red you have other issues to attend to.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.