# Unassigned shard on Logstash index rollover **URL:** \ **Category:** Elasticsearch\ **Created:** [February 17, 2014, 8:36am UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837 "2014-02-17T08:36:32Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![Neil\_Prosser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neil_prosser/32/1781_2.png) [@Neil\_Prosser](https://discuss.elastic.co/u/Neil_Prosser)\ **Post date:** [February 17, 2014, 8:36am UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/1 "2014-02-17T08:36:32Z")
I have a cluster with five nodes and I've specified that I'd like to allocate a maximum of two shards to each node for indexes which are created with five shards and two replicas of each (making ten shards to allocate in total for each index). Occasionally I find that, at midnight when the new index is created, a shard stays unassigned. For example, yesterday's index was allocated as follows (brackets indicate primary shard): node01 - (2) 3 node02 - 0 (4) node03 - 2 (3) node04 - (0) (1) node05 - 1 4 This morning the new index was allocated as follows: node01 - (2) 3 node02 - 2 (3) node03 - (4) node04 - 0 (1) node05 - (0) 1 unassigned - 4 I now have to go in and manually move a shard from one of the nodes and the shard is then allocated. The settings for the indices are: settings: { index.analysis.analyzer.url\_path\_analyzer.type: custom index.query.default\_field: message index.number\_of\_replicas: 1 index.number\_of\_shards: 5 index.auto\_expand\_replicas: false index.routing.allocation.total\_shards\_per\_node: 2 index.store.compress.tv: true index.analysis.tokenizer.url\_path\_tokenizer.type: path\_hierarchy index.store.compress.stored: true index.analysis.tokenizer.url\_path\_tokenizer.delimiter: / index.cache.field.type: soft index.analysis.analyzer.url\_path\_analyzer.tokenizer: url\_path\_tokenizer index.version.created: 901199 index.uuid: XRooj-ZmRe2c58uYDzsMFQ } It's not using the standard Logstash settings (Logstash's elasticsearch output is set to manage\_templates =\> false). Does anyone have any ideas as to what I've done wrong that is likely to be causing these issues? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/731c80d3-c6e0-4afe-b45d-92c70d774e2a%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/731c80d3-c6e0-4afe-b45d-92c70d774e2a%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [February 18, 2014, 2:56pm UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/2 "2014-02-18T14:56:33Z")
Very odd indeed. May I ask which version of ES are you using, and also how/where are you setting this: "index.routing.allocation.total\_shards\_per\_node: 2"? I want to see if I can duplicate your behavior. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/755a730f-e2da-49b9-b330-bfbba2bdf9b1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/755a730f-e2da-49b9-b330-bfbba2bdf9b1%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Neil\_Prosser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neil_prosser/32/1781_2.png) [@Neil\_Prosser](https://discuss.elastic.co/u/Neil_Prosser)\ **Post date:** [February 18, 2014, 7:11pm UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/3 "2014-02-18T19:11:23Z")
I'm using 0.90.11 but I have seen this behaviour in older versions. The total\_shards\_per\_node setting is set via a mapping file living in $ES\_HOME/config/templates/logstash.json. The relevant part of the mapping is: { "logstash" : { "template" : "logstash-\*", "settings" : { "index" : { ... "routing" : { "allocation" : { "total\_shards\_per\_node" : 2 } } } }, "mappings" : { ... } } } If you need any other information let me know. I still have the cluster in a yellow state. It also did the same thing this morning. node01 - 2 (3) node02 - (4) node03 - (0) 1 node04 - (2) 3 node05 - 0 (1) unassigned - 4 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6691e9c0-aff9-4fee-98f5-322bdeac7a75%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6691e9c0-aff9-4fee-98f5-322bdeac7a75%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Neil\_Prosser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neil_prosser/32/1781_2.png) [@Neil\_Prosser](https://discuss.elastic.co/u/Neil_Prosser)\ **Post date:** [February 20, 2014, 9:47am UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/4 "2014-02-20T09:47:33Z")
On the 19th it left the replica of shard 4 unassigned: node01 - (4) node02 - 0 (1) node03 - (0) 1 node04 - 2 (3) node05 - (2) 3 unassigned 4 This morning however, it's allocated all the shards: node01 - (0) 1 node02 - (2) 3 node03 - (1) 2 node04 - 0 (4) node05 - (3) 4 -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/c3e5861f-1b82-49d2-bcfa-8813fb3421ef%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/c3e5861f-1b82-49d2-bcfa-8813fb3421ef%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [February 20, 2014, 2:58pm UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/5 "2014-02-20T14:58:06Z")
Neil, I have been trying to reproduce but I can't seem to (0.90.11 and 1.0.0). Perhaps is it possible for you to look at your logs on all nodes and see if there is anything there that might pinpoint/relate to this? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f6471b74-4695-4661-92a9-bf796af48f0c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f6471b74-4695-4661-92a9-bf796af48f0c%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Neil\_Prosser](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/neil_prosser/32/1781_2.png) [@Neil\_Prosser](https://discuss.elastic.co/u/Neil_Prosser)\ **Post date:** [February 21, 2014, 7:51pm UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/6 "2014-02-21T19:51:38Z")
Sure, looking at the top of the logs for the 17th (when the problem started) I can see... [2014-02-17 00:00:00,278][INFO][cluster.metadata] [Caretaker] [logstash-2014.02.17] creating index, cause [auto(bulk api)], shards [5]/[1], mappings [_default_] [2014-02-17 00:00:00,336][WARN][transport.netty] [Caretaker] Message not fully read (request) for [5684538] and action [cluster/nodeIndexCreated], resetting [2014-02-17 00:00:00,336][WARN][transport.netty] [Caretaker] Message not fully read (request) for [5358797] and action [cluster/nodeIndexCreated], resetting [2014-02-17 00:00:00,336][WARN][transport.netty] [Caretaker] Message not fully read (request) for [6186213] and action [cluster/nodeIndexCreated], resetting ... before it gets into the usual stuff about updating dynamic mappings for the types I've got. It only occurs in the logs on one of the nodes and it's the node which is currently the master (and I haven't purposely changed that by restarting nodes or anything similar since then). The logs on the other nodes don't pick up until 09:00 when I was restarting the Logstash processes which had joined the cluster. That pattern of logs actually occurs every day when the new index is created (given the message content I'm assuming that's the case). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8a9970ce-563c-4dd5-a14b-55079018aedf%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8a9970ce-563c-4dd5-a14b-55079018aedf%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\ **Post date:** [February 21, 2014, 9:52pm UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/7 "2014-02-21T21:52:35Z")
Hmmm, I'm assuming you are running LS 1.3.3 and using the elasticsearch output. I'm wondering if you can use the elasticsearch\_http output instead and see if that makes any difference. I am very curious to know if it works or not. So something like this in the LS config: output { #elasticsearch { # host =\> "localhost" # port =\> 9300 #} elasticsearch\_http { host =\> "localhost" } } -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/48b1eaae-9796-42b5-841d-99002e286518%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/48b1eaae-9796-42b5-841d-99002e286518%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:48am UTC](https://discuss.elastic.co/t/unassigned-shard-on-logstash-index-rollover/15837/8 "2017-07-06T01:48:14Z")