# Stability problems while indexing with 0.18.4 **URL:** \ **Category:** Elasticsearch\ **Created:** [November 28, 2011, 9:12pm UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000 "2011-11-28T21:12:26Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [November 28, 2011, 9:12pm UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/1 "2011-11-28T21:12:26Z")
Hi, I am running a "staging" 6 nodes cluster which, for now, only indexes our realtime data, which is a duplicate of what we are indexing in production, roughly 10-15M small documents per day, stored in "daily" indices. Not much search load is performed on this cluster yet, mainly only indexing data, using the bulk api. Cluster is running 0.18.4, on EC2/Ubuntu 10.10 using OpenJDK Runtime Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.10.2) ( java version "1.6.0\_20") On this cluster all nodes are http+data and I setup nginx with reverse proxy to load balance all ES http queries in round robin across the cluster. Now, the problem I am seeing is yellow/red cluster health status, starting right after the new index creation at midnight. after that it get "stuck" at relocating\_shards, and initializing\_shards. To solve this I have to identify the node with the "stuck" shard and restart it. This solves the problem. Here's a theory: my daily indices are implicitly created when the first document with a date on the new day is indexed. I am not explicitly calling the create index api. Could there be some race condition with the implicit index creation when all nodes receive almost at the same time documents for a new inexistant index? (since I am load balancing all my http requests across all nodes on my cluster). I tried pre-creating the next index and the problem did not show up. I don't have much to back this theory, and there isn't any error log showing up to diagnose the problem. One thing I did this morning is add an extra http-only node, which I will use only to handle all my indexing queries. The search queries will be load balanced on the 6 data nodes. This is pretty-much the setup we currently have in production which has been working well so far. I figured funnelling all indexing queries through a single node might not be prone to my theoretical race condition. Any thoughts? Thanks, Colin ---
**Author:** ![AEvar\_Arnfjord\_Bjarm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/aevar_arnfjord_bjarm/32/2746_2.png) [@AEvar\_Arnfjord\_Bjarm](https://discuss.elastic.co/u/AEvar_Arnfjord_Bjarm)\ **Post date:** [November 28, 2011, 9:29pm UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/2 "2011-11-28T21:29:44Z")
On Nov 28, 2011 10:12 PM, "Colin Surprenant" [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > (since I > am load balancing all my http requests across all nodes on my > cluster). That sounds like it could cause problems. Why are you load balancing it in the first place instead of having the clients round-robin? ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [November 28, 2011, 9:47pm UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/3 "2011-11-28T21:47:22Z")
just a layering strategy, so that the client does not have to deal with doing round robin. changes in the topology does not require a client update. simply deal with it at nginx config. On Mon, Nov 28, 2011 at 4:29 PM, Ævar Arnfjörð Bjarmason [avarab@gmail.com](mailto:avarab@gmail.com) wrote: > On Nov 28, 2011 10:12 PM, "Colin Surprenant" [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) > wrote: > > > (since I > > am load balancing all my http requests across all nodes on my > > cluster). > > That sounds like it could cause problems. > > Why are you load balancing it in the first place instead of having the > clients round-robin? ---
**Author:** ![cameronnewman](https://avatars.discourse-cdn.com/v4/letter/c/90ced4/32.png) [@cameronnewman](https://discuss.elastic.co/u/cameronnewman)\ **Post date:** [November 29, 2011, 6:13am UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/4 "2011-11-29T06:13:08Z")
I'm experiencing a similar issue on 0.18.4. I also have daily indexes of about ~40 million documents with 16 shards/2 replicas per index across 8 nodes. The first couple of indexes work fine with indexing and searching, once I get to ~7-8 indexes things start to go funny. I normally find that 1-3 nodes have their CPU pegged at 100%. Each node is a VM running Centos 5.6 x64 4Gb and 4 vCPUs (on VMware). Any ideas? Should I not be having so many indexes? thanks ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 29, 2011, 9:27am UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/5 "2011-11-29T09:27:07Z")
Its perfectly fine to have a load balancer in front of elasticsearch nodes. When you execute the bulk API against the cluster, and an index needs to be created as a result of it, it will automatically execute the create index API and once its done, execute the bulk API. The create index API always executes on the elected master in the cluster. Its strange that it hangs, would love to see the log on the master node. I just ran a 6 node cluster and execute bulk indexing on each one concurrently indexing into a non existent index, and it worked well. How easy is it to recreate? Trying to understand whats going on is going to be a bit tricky. If you have time, I can create debug version of 0.18 and we can try and figure it out online on IRC. -shay.banon On Mon, Nov 28, 2011 at 11:12 PM, Colin Surprenant \< [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com)\> wrote: > Hi, > > I am running a "staging" 6 nodes cluster which, for now, only indexes > our realtime data, which is a duplicate of what we are indexing in > production, roughly 10-15M small documents per day, stored in "daily" > indices. Not much search load is performed on this cluster yet, mainly > only indexing data, using the bulk api. > > Cluster is running 0.18.4, on EC2/Ubuntu 10.10 using OpenJDK Runtime > Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.10.2) ( java > version "1.6.0\_20") > > On this cluster all nodes are http+data and I setup nginx with reverse > proxy to load balance all ES http queries in round robin across the > cluster. > > Now, the problem I am seeing is yellow/red cluster health status, > starting right after the new index creation at midnight. after that it > get "stuck" at relocating\_shards, and initializing\_shards. To solve > this I have to identify the node with the "stuck" shard and restart > it. This solves the problem. > > Here's a theory: my daily indices are implicitly created when the > first document with a date on the new day is indexed. I am not > explicitly calling the create index api. Could there be some race > condition with the implicit index creation when all nodes receive > almost at the same time documents for a new inexistant index? (since I > am load balancing all my http requests across all nodes on my > cluster). > > I tried pre-creating the next index and the problem did not show up. I > don't have much to back this theory, and there isn't any error log > showing up to diagnose the problem. > > One thing I did this morning is add an extra http-only node, which I > will use only to handle all my indexing queries. The search queries > will be load balanced on the 6 data nodes. This is pretty-much the > setup we currently have in production which has been working well so > far. I figured funnelling all indexing queries through a single node > might not be prone to my theoretical race condition. > > Any thoughts? > > Thanks, > Colin ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [November 29, 2011, 4:20pm UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/6 "2011-11-29T16:20:27Z")
I am going to spend some time on trying to reproduce the problem in a more systematic way, once I have this, we'll get in touch to figure out what's happening, hopefully later today. Do you suggest I upgrade to 0.18.5 before starting my tests? Thanks, Colin On Tue, Nov 29, 2011 at 4:27 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Its perfectly fine to have a load balancer in front of elasticsearch nodes. > When you execute the bulk API against the cluster, and an index needs to be > created as a result of it, it will automatically execute the create index > API and once its done, execute the bulk API. The create index API always > executes on the elected master in the cluster. > Its strange that it hangs, would love to see the log on the master node. I > just ran a 6 node cluster and execute bulk indexing on each one concurrently > indexing into a non existent index, and it worked well. > How easy is it to recreate? Trying to understand whats going on is going to > be a bit tricky. If you have time, I can create debug version of 0.18 and we > can try and figure it out online on IRC. > -shay.banon > > On Mon, Nov 28, 2011 at 11:12 PM, Colin Surprenant > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > Hi, > > > > I am running a "staging" 6 nodes cluster which, for now, only indexes > > our realtime data, which is a duplicate of what we are indexing in > > production, roughly 10-15M small documents per day, stored in "daily" > > indices. Not much search load is performed on this cluster yet, mainly > > only indexing data, using the bulk api. > > > > Cluster is running 0.18.4, on EC2/Ubuntu 10.10 using OpenJDK Runtime > > Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.10.2) ( java > > version "1.6.0\_20") > > > > On this cluster all nodes are http+data and I setup nginx with reverse > > proxy to load balance all ES http queries in round robin across the > > cluster. > > > > Now, the problem I am seeing is yellow/red cluster health status, > > starting right after the new index creation at midnight. after that it > > get "stuck" at relocating\_shards, and initializing\_shards. To solve > > this I have to identify the node with the "stuck" shard and restart > > it. This solves the problem. > > > > Here's a theory: my daily indices are implicitly created when the > > first document with a date on the new day is indexed. I am not > > explicitly calling the create index api. Could there be some race > > condition with the implicit index creation when all nodes receive > > almost at the same time documents for a new inexistant index? (since I > > am load balancing all my http requests across all nodes on my > > cluster). > > > > I tried pre-creating the next index and the problem did not show up. I > > don't have much to back this theory, and there isn't any error log > > showing up to diagnose the problem. > > > > One thing I did this morning is add an extra http-only node, which I > > will use only to handle all my indexing queries. The search queries > > will be load balanced on the 6 data nodes. This is pretty-much the > > setup we currently have in production which has been working well so > > far. I figured funnelling all indexing queries through a single node > > might not be prone to my theoretical race condition. > > > > Any thoughts? > > > > Thanks, > > Colin ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [November 29, 2011, 4:23pm UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/7 "2011-11-29T16:23:55Z")
Yea, if you can, supporting based on 0.18.5 will be simpler. On Tue, Nov 29, 2011 at 6:20 PM, Colin Surprenant \< [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com)\> wrote: > I am going to spend some time on trying to reproduce the problem in a > more systematic way, once I have this, we'll get in touch to figure > out what's happening, hopefully later today. > > Do you suggest I upgrade to 0.18.5 before starting my tests? > > Thanks, > Colin > > On Tue, Nov 29, 2011 at 4:27 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > Its perfectly fine to have a load balancer in front of elasticsearch > > nodes. > > When you execute the bulk API against the cluster, and an index needs to > > be > > created as a result of it, it will automatically execute the create index > > API and once its done, execute the bulk API. The create index API always > > executes on the elected master in the cluster. > > Its strange that it hangs, would love to see the log on the master node. > > I > > just ran a 6 node cluster and execute bulk indexing on each one > > concurrently > > indexing into a non existent index, and it worked well. > > How easy is it to recreate? Trying to understand whats going on is going > > to > > be a bit tricky. If you have time, I can create debug version of 0.18 > > and we > > can try and figure it out online on IRC. > > -shay.banon > > > > On Mon, Nov 28, 2011 at 11:12 PM, Colin Surprenant > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > Hi, > > > > > > I am running a "staging" 6 nodes cluster which, for now, only indexes > > > our realtime data, which is a duplicate of what we are indexing in > > > production, roughly 10-15M small documents per day, stored in "daily" > > > indices. Not much search load is performed on this cluster yet, mainly > > > only indexing data, using the bulk api. > > > > > > Cluster is running 0.18.4, on EC2/Ubuntu 10.10 using OpenJDK Runtime > > > Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.10.2) ( java > > > version "1.6.0\_20") > > > > > > On this cluster all nodes are http+data and I setup nginx with reverse > > > proxy to load balance all ES http queries in round robin across the > > > cluster. > > > > > > Now, the problem I am seeing is yellow/red cluster health status, > > > starting right after the new index creation at midnight. after that it > > > get "stuck" at relocating\_shards, and initializing\_shards. To solve > > > this I have to identify the node with the "stuck" shard and restart > > > it. This solves the problem. > > > > > > Here's a theory: my daily indices are implicitly created when the > > > first document with a date on the new day is indexed. I am not > > > explicitly calling the create index api. Could there be some race > > > condition with the implicit index creation when all nodes receive > > > almost at the same time documents for a new inexistant index? (since I > > > am load balancing all my http requests across all nodes on my > > > cluster). > > > > > > I tried pre-creating the next index and the problem did not show up. I > > > don't have much to back this theory, and there isn't any error log > > > showing up to diagnose the problem. > > > > > > One thing I did this morning is add an extra http-only node, which I > > > will use only to handle all my indexing queries. The search queries > > > will be load balanced on the 6 data nodes. This is pretty-much the > > > setup we currently have in production which has been working well so > > > far. I figured funnelling all indexing queries through a single node > > > might not be prone to my theoretical race condition. > > > > > > Any thoughts? > > > > > > Thanks, > > > Colin ---
**Author:** ![colinsurprenant](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/colinsurprenant/32/14776_2.png) [@colinsurprenant](https://discuss.elastic.co/u/colinsurprenant)\ **Post date:** [November 29, 2011, 8:55pm UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/8 "2011-11-29T20:55:26Z")
Damn it, can't reproduce the problem. I will restart the environment exactly like it was setup previously, but with DEBUG logging level. If/when it fails, we should have a better view of what's happening. Colin On Tue, Nov 29, 2011 at 11:23 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > Yea, if you can, supporting based on 0.18.5 will be simpler. > > On Tue, Nov 29, 2011 at 6:20 PM, Colin Surprenant > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > I am going to spend some time on trying to reproduce the problem in a > > more systematic way, once I have this, we'll get in touch to figure > > out what's happening, hopefully later today. > > > > Do you suggest I upgrade to 0.18.5 before starting my tests? > > > > Thanks, > > Colin > > > > On Tue, Nov 29, 2011 at 4:27 AM, Shay Banon [kimchy@gmail.com](mailto:kimchy@gmail.com) wrote: > > > > > Its perfectly fine to have a load balancer in front of elasticsearch > > > nodes. > > > When you execute the bulk API against the cluster, and an index needs to > > > be > > > created as a result of it, it will automatically execute the create > > > index > > > API and once its done, execute the bulk API. The create index API always > > > executes on the elected master in the cluster. > > > Its strange that it hangs, would love to see the log on the master node. > > > I > > > just ran a 6 node cluster and execute bulk indexing on each one > > > concurrently > > > indexing into a non existent index, and it worked well. > > > How easy is it to recreate? Trying to understand whats going on is going > > > to > > > be a bit tricky. If you have time, I can create debug version of 0.18 > > > and we > > > can try and figure it out online on IRC. > > > -shay.banon > > > > > > On Mon, Nov 28, 2011 at 11:12 PM, Colin Surprenant > > > [colin.surprenant@gmail.com](mailto:colin.surprenant@gmail.com) wrote: > > > > > > > Hi, > > > > > > > > I am running a "staging" 6 nodes cluster which, for now, only indexes > > > > our realtime data, which is a duplicate of what we are indexing in > > > > production, roughly 10-15M small documents per day, stored in "daily" > > > > indices. Not much search load is performed on this cluster yet, mainly > > > > only indexing data, using the bulk api. > > > > > > > > Cluster is running 0.18.4, on EC2/Ubuntu 10.10 using OpenJDK Runtime > > > > Environment (IcedTea6 1.9.10) (6b20-1.9.10-0ubuntu1~10.10.2) ( java > > > > version "1.6.0\_20") > > > > > > > > On this cluster all nodes are http+data and I setup nginx with reverse > > > > proxy to load balance all ES http queries in round robin across the > > > > cluster. > > > > > > > > Now, the problem I am seeing is yellow/red cluster health status, > > > > starting right after the new index creation at midnight. after that it > > > > get "stuck" at relocating\_shards, and initializing\_shards. To solve > > > > this I have to identify the node with the "stuck" shard and restart > > > > it. This solves the problem. > > > > > > > > Here's a theory: my daily indices are implicitly created when the > > > > first document with a date on the new day is indexed. I am not > > > > explicitly calling the create index api. Could there be some race > > > > condition with the implicit index creation when all nodes receive > > > > almost at the same time documents for a new inexistant index? (since I > > > > am load balancing all my http requests across all nodes on my > > > > cluster). > > > > > > > > I tried pre-creating the next index and the problem did not show up. I > > > > don't have much to back this theory, and there isn't any error log > > > > showing up to diagnose the problem. > > > > > > > > One thing I did this morning is add an extra http-only node, which I > > > > will use only to handle all my indexing queries. The search queries > > > > will be load balanced on the 6 data nodes. This is pretty-much the > > > > setup we currently have in production which has been working well so > > > > far. I figured funnelling all indexing queries through a single node > > > > might not be prone to my theoretical race condition. > > > > > > > > Any thoughts? > > > > > > > > Thanks, > > > > Colin ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:47am UTC](https://discuss.elastic.co/t/stability-problems-while-indexing-with-0-18-4/6000/9 "2017-07-06T03:47:01Z")