# .16.3 Hangs **URL:** \ **Category:** Elasticsearch\ **Created:** [July 15, 2011, 4:34am UTC](https://discuss.elastic.co/t/16-3-hangs/4868 "2011-07-15T04:34:16Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![electic](https://avatars.discourse-cdn.com/v4/letter/e/ecccb3/32.png) [@electic](https://discuss.elastic.co/u/electic)\ **Post date:** [July 15, 2011, 4:34am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/1 "2011-07-15T04:34:16Z")
I am still having trouble with Elastic Search. After a day of tens of thousands of inserts via the bulk api we notice that Elastic Search hangs. The REST api stops responding and the CPU is at 100 percent. The box has 16GB of Ram and a 7200 RPM disk. Is there some wehre I can look in the log about what is causing this? The logs show no exceptions. The only cooky thing we are doing is using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 15, 2011, 5:15am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/2 "2011-07-15T05:15:39Z")
Those are tricky to track down, lets start from the basics: 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? -shay.banon On Friday, July 15, 2011 at 7:34 AM, electic wrote: > I am still having trouble with Elastic Search. After a day of tens of > thousands of inserts via the bulk api we notice that Elastic Search > hangs. The REST api stops responding and the CPU is at 100 percent. > The box has 16GB of Ram and a 7200 RPM disk. > > Is there some wehre I can look in the log about what is causing this? > The logs show no exceptions. The only cooky thing we are doing is > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![electic](https://avatars.discourse-cdn.com/v4/letter/e/ecccb3/32.png) [@electic](https://discuss.elastic.co/u/electic)\ **Post date:** [July 15, 2011, 5:18am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/3 "2011-07-15T05:18:17Z")
Hi Shay, 1. Yep, we have two nodes. Exact same config. And it is our own hardware. 2. I have changed elasticsearch.in.sh with a 512MB with a max of 10GB. Here is the process running: 1636 pts/0 Sl 13:56 /usr/bin/java -Xms256m -Xmx10g -Xss128k - Djline.enabled=true -XX:+UseParNewGC 1. I am not sure what this means, can you elaborate. Happy to run any debug on our test machines. Raj On Jul 14, 10:15 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Those are tricky to track down, lets start from the basics: > > 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? > 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). > 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? > > -shay.banon > > On Friday, July 15, 2011 at 7:34 AM, electic wrote: > > > I am still having trouble with Elastic Search. After a day of tens of > > thousands of inserts via the bulk api we notice that Elastic Search > > hangs. The REST api stops responding and the CPU is at 100 percent. > > The box has 16GB of Ram and a 7200 RPM disk. > > > Is there some wehre I can look in the log about what is causing this? > > The logs show no exceptions. The only cooky thing we are doing is > > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [July 15, 2011, 5:31am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/4 "2011-07-15T05:31:20Z")
Heya, Use the same value for MIN and MAX, its recommended, especially if you use bootstrap.mlockall. Its explained here: [Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html). Also, I only see part of the process parameters, I assume you have the rest of the parameters set... When you run it, can you run bigdesk against it, check if its maxing out on memory? [GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) Last, can you gist your config? On Friday, July 15, 2011 at 8:18 AM, electic wrote: > Hi Shay, > > 1. Yep, we have two nodes. Exact same config. And it is our own > hardware. > 2. I have changed elasticsearch.in.sh ([http://elasticsearch.in.sh](http://elasticsearch.in.sh)) with a 512MB with a max of 10GB. > Here is the process running: > > 1636 pts/0 Sl 13:56 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > Djline.enabled=true -XX:+UseParNewGC > > 1. I am not sure what this means, can you elaborate. Happy to run any > debug on our test machines. > > Raj > > On Jul 14, 10:15 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > Those are tricky to track down, lets start from the basics: > > > > 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? > > 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). > > 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? > > > > -shay.banon > > > > On Friday, July 15, 2011 at 7:34 AM, electic wrote: > > > > > I am still having trouble with Elastic Search. After a day of tens of > > > thousands of inserts via the bulk api we notice that Elastic Search > > > hangs. The REST api stops responding and the CPU is at 100 percent. > > > The box has 16GB of Ram and a 7200 RPM disk. > > > > > Is there some wehre I can look in the log about what is causing this? > > > The logs show no exceptions. The only cooky thing we are doing is > > > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![electic](https://avatars.discourse-cdn.com/v4/letter/e/ecccb3/32.png) [@electic](https://discuss.elastic.co/u/electic)\ **Post date:** [July 15, 2011, 7:01am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/5 "2011-07-15T07:01:26Z")
Hi Shay, Sorry about that, pasting slops. Here it is again: 17879 ? Sl 47:26 /usr/bin/java -Xms256m -Xmx10g -Xss128k - Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: +CMSParallelRemarkEnabled -XX:SurvivorRatio=8 - XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX: +UseCMSInitiatingOccupancy Only -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=/ usr/local/elasticsearch -Des-pidfile=/var/run/elasticsearch.pid -cp ๐Ÿ˜• usr/local/elasticsearch/lib/elasticsearch-0.16.4.jar:/usr/local/ elasticsearch/lib/_:/usr/local/elasticsearch/lib/sigar/_ -Des.config=/ e tc/elasticsearch/elasticsearch.yml -Des.path.home=/usr/local/ elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/ var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch org.elasticsearch.bootstrap.Elasticsearch I will change the parameters and do the suggestions. Be back with my update. Thanks for your help! On Jul 14, 10:31 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Heya, > > Use the same value for MIN and MAX, its recommended, especially if you use bootstrap.mlockall. Its explained here:[Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html). > > Also, I only see part of the process parameters, I assume you have the rest of the parameters set... > > When you run it, can you run bigdesk against it, check if its maxing out on memory?[GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) > > Last, can you gist your config? > > On Friday, July 15, 2011 at 8:18 AM, electic wrote: > > > Hi Shay, > > > 1. Yep, we have two nodes. Exact same config. And it is our own > > hardware. > > 2. I have changed elasticsearch.in.sh ([http://elasticsearch.in.sh](http://elasticsearch.in.sh)) with a 512MB with a max of 10GB. > > Here is the process running: > > > 1636 pts/0 Sl 13:56 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > Djline.enabled=true -XX:+UseParNewGC > > > 1. I am not sure what this means, can you elaborate. Happy to run any > > debug on our test machines. > > > Raj > > > On Jul 14, 10:15 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > Those are tricky to track down, lets start from the basics: > > > > 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? > > > 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). > > > 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? > > > > -shay.banon > > > > On Friday, July 15, 2011 at 7:34 AM, electic wrote: > > > > > > > I am still having trouble with Elastic Search. After a day of tens of > > > > thousands of inserts via the bulk api we notice that Elastic Search > > > > hangs. The REST api stops responding and the CPU is at 100 percent. > > > > The box has 16GB of Ram and a 7200 RPM disk. > > > > > Is there some wehre I can look in the log about what is causing this? > > > > The logs show no exceptions. The only cooky thing we are doing is > > > > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![electic](https://avatars.discourse-cdn.com/v4/letter/e/ecccb3/32.png) [@electic](https://discuss.elastic.co/u/electic)\ **Post date:** [July 15, 2011, 7:46pm UTC](https://discuss.elastic.co/t/16-3-hangs/4868/6 "2011-07-15T19:46:25Z")
Upgraded to 16.4, same situation with min and max the same. Ran top on the box and it throws: On node 2: -bash: fork: Cannot allocate memory electic@search1:/usr/local/elasticsearch/bin$ top -bash: fork: Cannot allocate memory On node 1: PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 20852 root 20 0 10.7g 10g 10m S 100 69.3 182:55.07 java so pretty sure it is a memory problem. What would cause the memory to get exhausted like this? On Jul 15, 12:01 am, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > Hi Shay, > > Sorry about that, pasting slops. Here it is again: > > 17879 ? Sl 47:26 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: > +CMSParallelRemarkEnabled -XX:SurvivorRatio=8 - > XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX: > +UseCMSInitiatingOccupancy > Only -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=/ > usr/local/elasticsearch -Des-pidfile=/var/run/elasticsearch.pid -cp ๐Ÿ˜• > usr/local/elasticsearch/lib/elasticsearch-0.16.4.jar:/usr/local/ > elasticsearch/lib/_:/usr/local/elasticsearch/lib/sigar/_ -Des.config=/ > e > tc/elasticsearch/elasticsearch.yml -Des.path.home=/usr/local/ > elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/ > var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch > org.elasticsearch.bootstrap.Elasticsearch > > I will change the parameters and do the suggestions. Be back with my > update. Thanks for your help! > > On Jul 14, 10:31 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Heya, > > > Use the same value for MIN and MAX, its recommended, especially if you use bootstrap.mlockall. Its explained here:[Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html). > > > Also, I only see part of the process parameters, I assume you have the rest of the parameters set... > > > When you run it, can you run bigdesk against it, check if its maxing out on memory?[GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) > > > Last, can you gist your config? > > > On Friday, July 15, 2011 at 8:18 AM, electic wrote: > > > > > Hi Shay, > > > > 1. Yep, we have two nodes. Exact same config. And it is our own > > > hardware. > > > 2. I have changed elasticsearch.in.sh ([http://elasticsearch.in.sh](http://elasticsearch.in.sh)) with a 512MB with a max of 10GB. > > > Here is the process running: > > > > 1636 pts/0 Sl 13:56 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > > Djline.enabled=true -XX:+UseParNewGC > > > > 1. I am not sure what this means, can you elaborate. Happy to run any > > > debug on our test machines. > > > > Raj > > > > On Jul 14, 10:15 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > > > Those are tricky to track down, lets start from the basics: > > > > > 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? > > > > 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). > > > > 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? > > > > > -shay.banon > > > > > On Friday, July 15, 2011 at 7:34 AM, electic wrote: > > > > > > > > > I am still having trouble with Elastic Search. After a day of tens of > > > > > thousands of inserts via the bulk api we notice that Elastic Search > > > > > hangs. The REST api stops responding and the CPU is at 100 percent. > > > > > The box has 16GB of Ram and a 7200 RPM disk. > > > > > > Is there some wehre I can look in the log about what is causing this? > > > > > The logs show no exceptions. The only cooky thing we are doing is > > > > > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 16, 2011, 7:11pm UTC](https://discuss.elastic.co/t/16-3-hangs/4868/7 "2011-07-16T19:11:19Z")
Hi, it looks like you ran out of sockets, not RAM. Sockets use file handles. Check if the ES process is using the maximum number of open files setting of the OS. It should be in the order of some 10.000. If the value is ok, check your ES client code. Clients should reuse socket connections, otherwise unused socket connections will thrash the server and indexing will become slow. Do not repeat opening and closing connections when using bulk indexing. Jรถrg On Jul 15, 9:46 pm, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > Upgraded to 16.4, same situation with min and max the same. Ran top on > the box and it throws: > > On node 2: > -bash: fork: Cannot allocate memory > electic@search1:/usr/local/elasticsearch/bin$ top > -bash: fork: Cannot allocate memory > > On node 1: > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > COMMAND > 20852 root 20 0 10.7g 10g 10m S 100 69.3 182:55.07 > java > > so pretty sure it is a memory problem. What would cause the memory to > get exhausted like this? > > On Jul 15, 12:01 am, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > > > Hi Shay, > > > Sorry about that, pasting slops. Here it is again: > > > 17879 ? Sl 47:26 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: > > +CMSParallelRemarkEnabled -XX:SurvivorRatio=8 - > > XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX: > > +UseCMSInitiatingOccupancy > > Only -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=/ > > usr/local/elasticsearch -Des-pidfile=/var/run/elasticsearch.pid -cp ๐Ÿ˜• > > usr/local/elasticsearch/lib/elasticsearch-0.16.4.jar:/usr/local/ > > elasticsearch/lib/_:/usr/local/elasticsearch/lib/sigar/_ -Des.config=/ > > e > > tc/elasticsearch/elasticsearch.yml -Des.path.home=/usr/local/ > > elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/ > > var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch > > org.elasticsearch.bootstrap.Elasticsearch > > > I will change the parameters and do the suggestions. Be back with my > > update. Thanks for your help! > > > On Jul 14, 10:31 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > Heya, > > > > Use the same value for MIN and MAX, its recommended, especially if you use bootstrap.mlockall. Its explained here:[Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html). > > > > Also, I only see part of the process parameters, I assume you have the rest of the parameters set... > > > > When you run it, can you run bigdesk against it, check if its maxing out on memory?[GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) > > > > Last, can you gist your config? > > > > On Friday, July 15, 2011 at 8:18 AM, electic wrote: > > > > > > > Hi Shay, > > > > > 1. Yep, we have two nodes. Exact same config. And it is our own > > > > hardware. > > > > 2. I have changed elasticsearch.in.sh ([http://elasticsearch.in.sh](http://elasticsearch.in.sh)) with a 512MB with a max of 10GB. > > > > Here is the process running: > > > > > 1636 pts/0 Sl 13:56 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > > > Djline.enabled=true -XX:+UseParNewGC > > > > > 1. I am not sure what this means, can you elaborate. Happy to run any > > > > debug on our test machines. > > > > > Raj > > > > > On Jul 14, 10:15 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > > > > > Those are tricky to track down, lets start from the basics: > > > > > > 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? > > > > > 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). > > > > > 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? > > > > > > -shay.banon > > > > > > On Friday, July 15, 2011 at 7:34 AM, electic wrote: > > > > > > > > > > > I am still having trouble with Elastic Search. After a day of tens of > > > > > > thousands of inserts via the bulk api we notice that Elastic Search > > > > > > hangs. The REST api stops responding and the CPU is at 100 percent. > > > > > > The box has 16GB of Ram and a 7200 RPM disk. > > > > > > > Is there some wehre I can look in the log about what is causing this? > > > > > > The logs show no exceptions. The only cooky thing we are doing is > > > > > > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![electic](https://avatars.discourse-cdn.com/v4/letter/e/ecccb3/32.png) [@electic](https://discuss.elastic.co/u/electic)\ **Post date:** [July 17, 2011, 4:07am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/8 "2011-07-17T04:07:13Z")
I highly doubt its that. The servers are configured to reuse the socket and close it as soon as possible. It is also configured to the max setting as to the number of sockets that can be open. The bulk index is called rarely once we have enough records to send to the first node. However, I have been testing it and I have a theory. Basically there are two nodes in the cluster. If I just keep one node and and disconnect the other one...it runs fine. However, once the second node is activated it begins to show issues. The second node runs out of memory which then causes an adverse effect on the node 1 and the entire cluster goes down. I am not sure what is causing Node 2 to run out of memory yet. Regards, Raj On Jul 16, 12:11 pm, jprante [joergpra...@gmail.com](mailto:joergpra...@gmail.com) wrote: > Hi, > > it looks like you ran out of sockets, not RAM. Sockets use file > handles. Check if the ES process is using the maximum number of open > files setting of the OS. It should be in the order of some 10.000. If > the value is ok, check your ES client code. Clients should reuse > socket connections, otherwise unused socket connections will thrash > the server and indexing will become slow. Do not repeat opening and > closing connections when using bulk indexing. > > Jรถrg > > On Jul 15, 9:46 pm, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > > > Upgraded to 16.4, same situation with min and max the same. Ran top on > > the box and it throws: > > > On node 2: > > -bash: fork: Cannot allocate memory > > electic@search1:/usr/local/elasticsearch/bin$ top > > -bash: fork: Cannot allocate memory > > > On node 1: > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > COMMAND > > 20852 root 20 0 10.7g 10g 10m S 100 69.3 182:55.07 > > java > > > so pretty sure it is a memory problem. What would cause the memory to > > get exhausted like this? > > > On Jul 15, 12:01 am, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > > > > Hi Shay, > > > > Sorry about that, pasting slops. Here it is again: > > > > 17879 ? Sl 47:26 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > > Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: > > > +CMSParallelRemarkEnabled -XX:SurvivorRatio=8 - > > > XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX: > > > +UseCMSInitiatingOccupancy > > > Only -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=/ > > > usr/local/elasticsearch -Des-pidfile=/var/run/elasticsearch.pid -cp ๐Ÿ˜• > > > usr/local/elasticsearch/lib/elasticsearch-0.16.4.jar:/usr/local/ > > > elasticsearch/lib/_:/usr/local/elasticsearch/lib/sigar/_ -Des.config=/ > > > e > > > tc/elasticsearch/elasticsearch.yml -Des.path.home=/usr/local/ > > > elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/ > > > var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch > > > org.elasticsearch.bootstrap.Elasticsearch > > > > I will change the parameters and do the suggestions. Be back with my > > > update. Thanks for your help! > > > > On Jul 14, 10:31 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > Heya, > > > > > Use the same value for MIN and MAX, its recommended, especially if you use bootstrap.mlockall. Its explained here:[Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html). > > > > > Also, I only see part of the process parameters, I assume you have the rest of the parameters set... > > > > > When you run it, can you run bigdesk against it, check if its maxing out on memory?[GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) > > > > > Last, can you gist your config? > > > > > On Friday, July 15, 2011 at 8:18 AM, electic wrote: > > > > > > > > > Hi Shay, > > > > > > 1. Yep, we have two nodes. Exact same config. And it is our own > > > > > hardware. > > > > > 2. I have changed elasticsearch.in.sh ([http://elasticsearch.in.sh](http://elasticsearch.in.sh)) with a 512MB with a max of 10GB. > > > > > Here is the process running: > > > > > > 1636 pts/0 Sl 13:56 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > > > > Djline.enabled=true -XX:+UseParNewGC > > > > > > 1. I am not sure what this means, can you elaborate. Happy to run any > > > > > debug on our test machines. > > > > > > Raj > > > > > > On Jul 14, 10:15 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > > > > > > > Those are tricky to track down, lets start from the basics: > > > > > > > 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? > > > > > > 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). > > > > > > 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? > > > > > > > -shay.banon > > > > > > > On Friday, July 15, 2011 at 7:34 AM, electic wrote: > > > > > > > > > > > > > I am still having trouble with Elastic Search. After a day of tens of > > > > > > > thousands of inserts via the bulk api we notice that Elastic Search > > > > > > > hangs. The REST api stops responding and the CPU is at 100 percent. > > > > > > > The box has 16GB of Ram and a 7200 RPM disk. > > > > > > > > Is there some wehre I can look in the log about what is causing this? > > > > > > > The logs show no exceptions. The only cooky thing we are doing is > > > > > > > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [July 17, 2011, 9:10am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/9 "2011-07-17T09:10:50Z")
The server should not close sockets as soon as possible. Just tell your client to reuse socket connections. All that I can say that I once encountered exactly the same challenge. After all, the second node didn't get the max file setting by accident. After I fixed that, everything was fine. But, if you submit large amounts of simultaneous bulk requests, the sooner you get the caches filled and several Lucene indexer background processes start off. This consumes a lot of system resources at a sudden, files and sockets. For this, I had to limit the number of parallel bulk requests at client side. There is a "sweet spot" between client bulk index pressure and available network/OS capacity. Currently, I submit 30 bulk requests in parallel at most. If the limit is reached, which happens every few minutes, the bulk indexing client have to wait for the server to finish previous requests, which can take some time. Here, it may take up to 20 seconds or so, it depends on the work the Lucene indexers must do. Jรถrg On 17 Jul., 06:07, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > I highly doubt its that. The servers are configured to reuse the > socket and close it as soonas possible. It is also configured to the > max setting as to the number of sockets that can be open. The bulk > index is called rarely once we have enough records to send to the > first node. However, I have been testing it and I have a theory. > > Basically there are two nodes in the cluster. If I just keep one node > and and disconnect the other one...it runs fine. However, once the > second node is activated it begins to show issues. The second node > runs out of memory which then causes an adverse effect on the node 1 > and the entire cluster goes down. I am not sure what is causing Node 2 > to run out of memory yet. > > Regards, > > Raj > > On Jul 16, 12:11 pm, jprante [joergpra...@gmail.com](mailto:joergpra...@gmail.com) wrote: > > > Hi, > > > it looks like you ran out of sockets, not RAM. Sockets use file > > handles. Check if the ES process is using the maximum number of open > > files setting of the OS. It should be in the order of some 10.000. If > > the value is ok, check your ES client code. Clients should reuse > > socket connections, otherwise unused socket connections will thrash > > the server and indexing will become slow. Do not repeat opening and > > closing connections when using bulk indexing. > > > Jรถrg > > > On Jul 15, 9:46 pm, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > > > > Upgraded to 16.4, same situation with min and max the same. Ran top on > > > the box and it throws: > > > > On node 2: > > > -bash: fork: Cannot allocate memory > > > electic@search1:/usr/local/elasticsearch/bin$ top > > > -bash: fork: Cannot allocate memory > > > > On node 1: > > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ > > > COMMAND > > > 20852 root 20 0 10.7g 10g 10m S 100 69.3 182:55.07 > > > java > > > > so pretty sure it is a memory problem. What would cause the memory to > > > get exhausted like this? > > > > On Jul 15, 12:01 am, electic [elec...@gmail.com](mailto:elec...@gmail.com) wrote: > > > > > Hi Shay, > > > > > Sorry about that, pasting slops. Here it is again: > > > > > 17879 ? Sl 47:26 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > > > Djline.enabled=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX: > > > > +CMSParallelRemarkEnabled -XX:SurvivorRatio=8 - > > > > XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX: > > > > +UseCMSInitiatingOccupancy > > > > Only -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.path.home=/ > > > > usr/local/elasticsearch -Des-pidfile=/var/run/elasticsearch.pid -cp ๐Ÿ˜• > > > > usr/local/elasticsearch/lib/elasticsearch-0.16.4.jar:/usr/local/ > > > > elasticsearch/lib/_:/usr/local/elasticsearch/lib/sigar/_ -Des.config=/ > > > > e > > > > tc/elasticsearch/elasticsearch.yml -Des.path.home=/usr/local/ > > > > elasticsearch -Des.path.logs=/var/log/elasticsearch -Des.path.data=/ > > > > var/lib/elasticsearch -Des.path.work=/tmp/elasticsearch > > > > org.elasticsearch.bootstrap.Elasticsearch > > > > > I will change the parameters and do the suggestions. Be back with my > > > > update. Thanks for your help! > > > > > On Jul 14, 10:31 pm, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > > > > Heya, > > > > > > Use the same value for MIN and MAX, its recommended, especially if you use bootstrap.mlockall. Its explained here:[Elasticsearch Platform โ€” Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/setup/installation.html). > > > > > > Also, I only see part of the process parameters, I assume you have the rest of the parameters set... > > > > > > When you run it, can you run bigdesk against it, check if its maxing out on memory?[GitHub - lukas-vlcek/bigdesk: Live charts and statistics for Elasticsearch cluster.](https://github.com/lukas-vlcek/bigdesk) > > > > > > Last, can you gist your config? > > > > > > On Friday, July 15, 2011 at 8:18 AM, electic wrote: > > > > > > > > > > > Hi Shay, > > > > > > > 1. Yep, we have two nodes. Exact same config. And it is our own > > > > > > hardware. > > > > > > 2. I have changed elasticsearch.in.sh ([http://elasticsearch.in.sh](http://elasticsearch.in.sh)) with a 512MB with a max of 10GB. > > > > > > Here is the process running: > > > > > > > 1636 pts/0 Sl 13:56 /usr/bin/java -Xms256m -Xmx10g -Xss128k - > > > > > > Djline.enabled=true -XX:+UseParNewGC > > > > > > > 1. I am not sure what this means, can you elaborate. Happy to run any > > > > > > debug on our test machines. > > > > > > > Raj > > > > > > > On Jul 14, 10:15 pm, Shay Banon \<[shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) ([http://elasticsearch.com](http://elasticsearch.com))\> wrote: > > > > > > > > > > > > > Those are tricky to track down, lets start from the basics: > > > > > > > > 1. I assume you run on your own hardware based on the configuration you posted? Or are you running on a virtualized system? Which operating system are you using? > > > > > > > 2. Maybe you get into memory problems with ES? How much memory are you allocating to elasticsearch? Can you run bigdesk against it and see how memory behaves? (upgrade to 0.16.4 if it happens). > > > > > > > 3. Are you using the bootstrap.mlockall option? If not, can you use it (just to make sure its not swapping)? > > > > > > > > -shay.banon > > > > > > > > On Friday, July 15, 2011 at 7:34 AM, electic wrote: > > > > > > > > > > > > > > > I am still having trouble with Elastic Search. After a day of tens of > > > > > > > > thousands of inserts via the bulk api we notice that Elastic Search > > > > > > > > hangs. The REST api stops responding and the CPU is at 100 percent. > > > > > > > > The box has 16GB of Ram and a 7200 RPM disk. > > > > > > > > > Is there some wehre I can look in the log about what is causing this? > > > > > > > > The logs show no exceptions. The only cooky thing we are doing is > > > > > > > > using the ID field to ensure uniqueness across the cluster. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:00am UTC](https://discuss.elastic.co/t/16-3-hangs/4868/10 "2017-07-06T04:00:23Z")