# BroadcastShardOperationFailedException **URL:** \ **Category:** Elasticsearch\ **Created:** [October 7, 2010, 8:22pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422 "2010-10-07T20:22:42Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [October 7, 2010, 8:22pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/1 "2010-10-07T20:22:42Z")
I am getting a BroadcastShardOperationFailedException when I do a GET against my cluster of two: { ``` * count: 0 * - _shards: { o total: 5 o successful: 3 o failed: 2 o - failures: [ + - { # index: "index0" # shard: 0 # reason: "BroadcastShardOperationFailedException[[index0][0] No active shard(s)]" } + - { # index: "index0" # shard: 2 # reason: "BroadcastShardOperationFailedException[[index0][2] No active shard(s)]" } ] } ``` } There should be 5 shards; I've restarted a number of times, and sometimes I get all 5 shards failing in essentially the same JSON as below (except it lists all shards). In one node log I see: [20:10:59,177][INFO][cluster.metadata] [Keith Kilham] [index0] creating index, cause [gateway], shards [5]/[1], mappings [mail, calendarItem, attachment, contact] [20:11:00,005][INFO][cluster.service] [Keith Kilham] added {[Tinkerer][e101b4b3-daa2-45fc-b042-ef14a5a56bea][inet[/10.244.255.242:9300]]{client=true, data=false, zen.master=false},}, reason: zen-disco-receive(from node[[Tinkerer][e101b4b3-daa2-45fc-b042-ef14a5a56bea][inet[/10.244.255.242:9300]]{client=true, data=false, zen.master=false}]) [20:11:00,039][INFO][cluster.service] [Keith Kilham] added {[Crystal][b6b67cac-650d-46e6-b949-a6497f2d8f81][inet[/10.244.255.242:9301]]{client=true, data=false, zen.master=false},}, reason: zen-disco-receive(from node[[Crystal][b6b67cac-650d-46e6-b949-a6497f2d8f81][inet[/10.244.255.242:9301]]{client=true, data=false, zen.master=false}]) [20:11:27,547][INFO][cluster.service] [Keith Kilham] added {[Madelyne Pryor][c3e71382-a3c6-4c7e-bc9c-283fa9fff23a][inet[/10.198.109.171:9300]],}, reason: zen-disco-receive(from node[[Madelyne Pryor][c3e71382-a3c6-4c7e-bc9c-283fa9fff23a][inet[/10.198.109.171:9300]]]) [20:11:27,588][INFO][http] [Keith Kilham] bound\_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish\_address {inet[/10.206.98.255:9200]} [20:11:27,732][INFO][jmx] [Keith Kilham] bound\_address {service:jmx:rmi:///jndi/rmi://:9400/jmxrmi}, publish\_address {service:jmx:rmi:///jndi/rmi://10.206.98.255:9400/jmxrmi} [20:11:27,733][INFO][node] [Keith Kilham] {elasticsearch/0.9.0}[11486]: started [20:11:31,139][WARN][index.gateway.s3] [Keith Kilham] [index0][0] no file [\_6vm.cfs] to recover, even though it has md5, ignoring it [20:11:31,140][WARN][index.gateway.s3] [Keith Kilham] [index0][0] no file [\_9bz.cfs] to recover, even though it has md5, ignoring it In the other node log I see scrolling by over an over constantly: [20:17:26,004][DEBUG][index.shard.service] [Madelyne Pryor] [index0][1] state: [RECOVERING]-\>[CREATED], restored after recovery [20:17:26,047][DEBUG][index.shard.service] [Madelyne Pryor] [index0][4] state: [CREATED]-\>[RECOVERING] [20:17:26,048][DEBUG][index.shard.service] [Madelyne Pryor] [index0][4] state: [RECOVERING]-\>[CREATED], restored after recovery [20:17:26,061][DEBUG][index.shard.service] [Madelyne Pryor] [index0][3] state: [CREATED]-\>[RECOVERING] [20:17:26,062][DEBUG][index.shard.service] [Madelyne Pryor] [index0][3] state: [RECOVERING]-\>[CREATED], restored after recovery [20:17:26,106][DEBUG][index.shard.service] [Madelyne Pryor] [index0][1] state: [CREATED]-\>[RECOVERING] [20:17:26,107][DEBUG][index.shard.service] [Madelyne Pryor] [index0][1] state: [RECOVERING]-\>[CREATED], restored after recovery When I write above of a 'cluster of two' I mean there are two data-carrying nodes. The third node, which is a no-data client only using the Java API, gets a PrimaryNotStartedActionException. I am on Elastic Search 0.9. It all used to work and started failing today. Yesterday I did call the REST optimize command and I saw these problems today after a restart (the first restart since the optimize). This is deployed in a shared environment and I can't upgrade to the latest Elastic Search right away, although I plan to when I can. Thanks for any help you can provide. ---
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [October 7, 2010, 8:46pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/2 "2010-10-07T20:46:43Z")
What it looks like when all the shards fail after a restart: { ``` * count: 0 * - _shards: { o total: 5 o successful: 0 o failed: 5 o - failures: [ + - { # index: "index0" # shard: 0 # reason: "BroadcastShardOperationFailedException[[index0][0] No active shard(s)]" } + - { # index: "index0" # shard: 2 # reason: "BroadcastShardOperationFailedException[[index0][2] No active shard(s)]" } + - { # index: "index0" # shard: 1 # reason: "BroadcastShardOperationFailedException[[index0][1] No active shard(s)]" } + - { # index: "index0" # shard: 4 # reason: "BroadcastShardOperationFailedException[[index0][4] No active shard(s)]" } + - { # index: "index0" # shard: 3 # reason: "BroadcastShardOperationFailedException[[index0][3] No active shard(s)]" } ] } ``` } ---
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [October 7, 2010, 9:21pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/3 "2010-10-07T21:21:39Z")
More info in case it is useful. Calling \_status on one of the document types yields a NoShardAvailableActionException: [http://204.236.195.66:9200/index0/mail/\_status](http://204.236.195.66:9200/index0/mail/_status) { ``` * error: "NoShardAvailableActionException[[index0][4] No shard available for [mail#_status]]" ``` } However, calling \_status on the index, it does know about at least some of the shards (there should be more, though, and there should be more documents, I'd expect -- I have 5 shards w/ a rep factor of 2, and it can only find 2): [http://204.236.195.66:9200/index0/\_status](http://204.236.195.66:9200/index0/_status) { ``` * ok: true * - _shards: { o total: 10 o successful: 2 o failed: 0 } * - indices: { o - index0: { + aliases: [] + - settings: { # index.number_of_replicas: "1" # index.number_of_shards: "5" } + store_size: "975.4mb" + store_size_in_bytes: 1022851097 + estimated_flushable_memory_size: "0b" + estimated_flushable_memory_size_in_bytes: 0 + translog_operations: 65 + - docs: { # num_docs: 640340 # max_doc: 647256 # deleted_docs: 6916 } + - shards: { # - 1: [ * - { o - routing: { + state: "STARTED" + primary: true + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" + relocating_node: null + shard: 1 + index: "index0" } o state: "STARTED" o store_size: "489.9mb" o store_size_in_bytes: 513748296 o estimated_flushable_memory_size: "0b" o estimated_flushable_memory_size_in_bytes: 0 o translog_id: 1285874875878 o translog_operations: 33 o - docs: { + num_docs: 320026 + max_doc: 323797 + deleted_docs: 3771 } } ] # - 3: [ * - { o - routing: { + state: "STARTED" + primary: true + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" + relocating_node: null + shard: 3 + index: "index0" } o state: "STARTED" o store_size: "485.5mb" o store_size_in_bytes: 509102801 o estimated_flushable_memory_size: "0b" o estimated_flushable_memory_size_in_bytes: 0 o translog_id: 1285874875925 o translog_operations: 32 o - docs: { + num_docs: 320314 + max_doc: 323459 + deleted_docs: 3145 } } ] } } } ``` } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 7, 2010, 9:25pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/4 "2010-10-07T21:25:07Z")
Can you run the cluster state API? Lets see where shards are allocated and at what state they are? On Thu, Oct 7, 2010 at 11:21 PM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > More info in case it is useful. > > Calling \_status on one of the document types yields a > NoShardAvailableActionException: > > [http://204.236.195.66:9200/index0/mail/\_status](http://204.236.195.66:9200/index0/mail/_status) > > { > > - error: "NoShardAvailableActionException[[index0][4] No shard available > for [mail#\_status]]" > > } > > However, calling \_status on the index, it does know about at least some of > the shards (there should be more, though, and there should be more > documents, I'd expect -- I have 5 shards w/ a rep factor of 2, and it can > only find 2): > > [http://204.236.195.66:9200/index0/\_status](http://204.236.195.66:9200/index0/_status) > > { > > - ok: true > - > - > > \_shards: { > o total: 10 > o successful: 2 > o failed: 0 > } > - > - > > indices: { > o > - > index0: { > + aliases: > + > - > settings: { > # index.number\_of\_replicas: "1" > # index.number\_of\_shards: "5" > } > + store\_size: "975.4mb" > + store\_size\_in\_bytes: 1022851097 > + estimated\_flushable\_memory\_size: "0b" > + estimated\_flushable\_memory\_size\_in\_bytes: 0 > + translog\_operations: 65 > + > - > docs: { > # num\_docs: 640340 > # max\_doc: 647256 > # deleted\_docs: 6916 > } > + > - > shards: { > # > - > 1: [ > \* > - > { > o > - > routing: { > + state: "STARTED" > + primary: true > + node: > "cf1b1780-3d04-4e09-b1dc-f55707acded2" > + relocating\_node: null > + shard: 1 > + index: "index0" > } > o state: "STARTED" > o store\_size: "489.9mb" > o store\_size\_in\_bytes: 513748296 > o estimated\_flushable\_memory\_size: "0b" > o > estimated\_flushable\_memory\_size\_in\_bytes: 0 > o translog\_id: 1285874875878 > o translog\_operations: 33 > o > - > docs: { > + num\_docs: 320026 > + max\_doc: 323797 > + deleted\_docs: 3771 > } > } > ] > # > - > 3: [ > \* > - > { > o > - > routing: { > + state: "STARTED" > + primary: true > + node: > "cf1b1780-3d04-4e09-b1dc-f55707acded2" > + relocating\_node: null > + shard: 3 > + index: "index0" > } > o state: "STARTED" > o store\_size: "485.5mb" > o store\_size\_in\_bytes: 509102801 > o estimated\_flushable\_memory\_size: "0b" > o > estimated\_flushable\_memory\_size\_in\_bytes: 0 > o translog\_id: 1285874875925 > o translog\_operations: 32 > o > - > docs: { > + num\_docs: 320314 > + max\_doc: 323459 > + deleted\_docs: 3145 > } > } > ] > } > } > } > > ## } > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1650984.html](http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1650984.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [October 7, 2010, 9:42pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/5 "2010-10-07T21:42:01Z")
[http://204.236.195.66:9200/\_cluster/state?pretty=true](http://204.236.195.66:9200/_cluster/state?pretty=true) produces: { ``` * cluster_name: "dev5" * master_node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" * blocks: { } * - nodes: { o - 4dac812b-f9df-4ecf-b0f3-e4a159d9189b: { + name: "Sagittarius" + transport_address: "inet[/10.198.109.171:9300]" } o - b6b67cac-650d-46e6-b949-a6497f2d8f81: { + name: "Crystal" + transport_address: "inet[/10.244.255.242:9301]" } o - e101b4b3-daa2-45fc-b042-ef14a5a56bea: { + name: "Tinkerer" + transport_address: "inet[/10.244.255.242:9300]" } o - cf1b1780-3d04-4e09-b1dc-f55707acded2: { + name: "Wolfsbane" + transport_address: "inet[/10.206.98.255:9300]" } } * - metadata: { o - indices: { + - index0: { # - settings: { * index.number_of_replicas: "1" * index.number_of_shards: "5" } # - mappings: { * - mail: { o - _boost: { + name: "_boost" } o dynamic: true o enabled: true o - date_formats: [ + "dateOptionalTime" + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" ] o - _source: { + enabled: false + name: "_source" } o - _id: { + store: "no" } o path: "full" o - properties: { + - to: { # omit_term_freq_and_positions: false # index_name: "to" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - body: { # omit_term_freq_and_positions: false # index_name: "body" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - receivedDateAsFloat: { # omit_term_freq_and_positions: false # index_name: "receivedDateAsFloat" # index: "no" # omit_norms: false # store: "yes" # boost: 1 # precision_step: 4 # term_vector: "no" # type: "float" } + - documentType: { # omit_term_freq_and_positions: false # index_name: "documentType" # index: "analyzed" # omit_norms: false # store: "no" # boost: 1 # term_vector: "no" # type: "string" } + - mailId: { # omit_term_freq_and_positions: false # index_name: "mailId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - subject: { # omit_term_freq_and_positions: false # index_name: "subject" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - userId: { # omit_term_freq_and_positions: false # index_name: "userId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - from: { # omit_term_freq_and_positions: false # index_name: "from" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - cc: { # omit_term_freq_and_positions: false # index_name: "cc" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - receivedDate: { # omit_term_freq_and_positions: false # index_name: "receivedDate" # index: "no" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } } o - _all: { + enabled: true + store: "no" + term_vector: "no" } o type: "object" } * - calendarItem: { o - _boost: { + name: "_boost" } o dynamic: true o enabled: true o - date_formats: [ + "dateOptionalTime" + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" ] o - _source: { + enabled: false + name: "_source" } o - _id: { + store: "no" } o path: "full" o - properties: { + - attendees: { # omit_term_freq_and_positions: false # index_name: "attendees" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - title: { # omit_term_freq_and_positions: false # index_name: "title" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - documentType: { # omit_term_freq_and_positions: false # index_name: "documentType" # index: "analyzed" # omit_norms: false # store: "no" # boost: 1 # term_vector: "no" # type: "string" } + - details: { # omit_term_freq_and_positions: false # index_name: "details" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - location: { # omit_term_freq_and_positions: false # index_name: "location" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - start: { # omit_term_freq_and_positions: false # index_name: "start" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - userId: { # omit_term_freq_and_positions: false # index_name: "userId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - organizer: { # omit_term_freq_and_positions: false # index_name: "organizer" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - calendarItemId: { # omit_term_freq_and_positions: false # index_name: "calendarItemId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } } o - _all: { + enabled: true + store: "no" + term_vector: "no" } o type: "object" } * - attachment: { o - _boost: { + name: "_boost" } o dynamic: true o enabled: true o - date_formats: [ + "dateOptionalTime" + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" ] o - _source: { + enabled: false + name: "_source" } o - _id: { + store: "no" } o path: "full" o - properties: { + - attachedDate: { # omit_term_freq_and_positions: false # index_name: "attachedDate" # index: "no" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - receivedDateAsFloat: { # omit_term_freq_and_positions: false # index_name: "receivedDateAsFloat" # index: "no" # omit_norms: false # store: "yes" # boost: 1 # precision_step: 4 # term_vector: "no" # type: "float" } + - documentType: { # omit_term_freq_and_positions: false # index_name: "documentType" # index: "analyzed" # omit_norms: false # store: "no" # boost: 1 # term_vector: "no" # type: "string" } + - attachmentId: { # omit_term_freq_and_positions: false # index_name: "attachmentId" # index: "no" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - mailId: { # omit_term_freq_and_positions: false # index_name: "mailId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - userId: { # omit_term_freq_and_positions: false # index_name: "userId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - fileName: { # omit_term_freq_and_positions: false # index_name: "fileName" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } } o - _all: { + enabled: true + store: "no" + term_vector: "no" } o type: "object" } * - contact: { o - _boost: { + name: "_boost" } o dynamic: true o enabled: true o - date_formats: [ + "dateOptionalTime" + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" ] o - _source: { + enabled: false + name: "_source" } o - _id: { + store: "no" } o path: "full" o - properties: { + - contactId: { # omit_term_freq_and_positions: false # index_name: "contactId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - documentType: { # omit_term_freq_and_positions: false # index_name: "documentType" # index: "analyzed" # omit_norms: false # store: "no" # boost: 1 # term_vector: "no" # type: "string" } + - organizations: { # omit_term_freq_and_positions: false # index_name: "organizations" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - userId: { # omit_term_freq_and_positions: false # index_name: "userId" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - fullName: { # omit_term_freq_and_positions: false # index_name: "fullName" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - emails: { # omit_term_freq_and_positions: false # index_name: "emails" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "yes" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - notes: { # omit_term_freq_and_positions: false # index_name: "notes" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - addresses: { # omit_term_freq_and_positions: false # index_name: "addresses" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } + - phones: { # omit_term_freq_and_positions: false # index_name: "phones" # index: "analyzed" # index_analyzer: "kihaSnowball" # omit_norms: false # store: "no" # search_analyzer: "kihaSnowball" # boost: 1 # term_vector: "no" # type: "string" } } o - _all: { + enabled: true + store: "no" + term_vector: "no" } o type: "object" } } # aliases: [] } } } * - routing_table: { o - indices: { + - index0: { # - shards: { * - 0: [ o - { + state: "INITIALIZING" + primary: true + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" + relocating_node: null + shard: 0 + index: "index0" } o - { + state: "UNASSIGNED" + primary: false + node: null + relocating_node: null + shard: 0 + index: "index0" } ] * - 1: [ o - { + state: "INITIALIZING" + primary: false + node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" + relocating_node: null + shard: 1 + index: "index0" } o - { + state: "STARTED" + primary: true + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" + relocating_node: null + shard: 1 + index: "index0" } ] * - 2: [ o - { + state: "INITIALIZING" + primary: true + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" + relocating_node: null + shard: 2 + index: "index0" } o - { + state: "UNASSIGNED" + primary: false + node: null + relocating_node: null + shard: 2 + index: "index0" } ] * - 3: [ o - { + state: "INITIALIZING" + primary: false + node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" + relocating_node: null + shard: 3 + index: "index0" } o - { + state: "STARTED" + primary: true + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" + relocating_node: null + shard: 3 + index: "index0" } ] * - 4: [ o - { + state: "INITIALIZING" + primary: true + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" + relocating_node: null + shard: 4 + index: "index0" } o - { + state: "UNASSIGNED" + primary: false + node: null + relocating_node: null + shard: 4 + index: "index0" } ] } } } } * - routing_nodes: { o - unassigned: [ + - { # state: "UNASSIGNED" # primary: false # node: null # relocating_node: null # shard: 0 # index: "index0" } + - { # state: "UNASSIGNED" # primary: false # node: null # relocating_node: null # shard: 2 # index: "index0" } + - { # state: "UNASSIGNED" # primary: false # node: null # relocating_node: null # shard: 4 # index: "index0" } ] o - nodes: { + - 4dac812b-f9df-4ecf-b0f3-e4a159d9189b: [ # - { * state: "INITIALIZING" * primary: false * node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" * relocating_node: null * shard: 1 * index: "index0" } # - { * state: "INITIALIZING" * primary: false * node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" * relocating_node: null * shard: 3 * index: "index0" } ] + - cf1b1780-3d04-4e09-b1dc-f55707acded2: [ # - { * state: "INITIALIZING" * primary: true * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" * relocating_node: null * shard: 0 * index: "index0" } # - { * state: "STARTED" * primary: true * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" * relocating_node: null * shard: 1 * index: "index0" } # - { * state: "INITIALIZING" * primary: true * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" * relocating_node: null * shard: 2 * index: "index0" } # - { * state: "STARTED" * primary: true * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" * relocating_node: null * shard: 3 * index: "index0" } # - { * state: "INITIALIZING" * primary: true * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" * relocating_node: null * shard: 4 * index: "index0" } ] } } ``` } ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 7, 2010, 9:44pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/6 "2010-10-07T21:44:32Z")
Seems like the shards are in initializing state, maybe it just takes time to recover from the shared storage? If you don't see exceptions, this is what they do... There were some bugs around it in 0.9, not sure if you hit one of them or not.... . -shay.banon p.s. Can you gist the info next time? Hard to make sense of it in mails... On Thu, Oct 7, 2010 at 11:42 PM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > [http://204.236.195.66:9200/\_cluster/state?pretty=true](http://204.236.195.66:9200/_cluster/state?pretty=true) > > produces: > > { > > - cluster\_name: "dev5" > - master\_node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" > - blocks: { } > - > - > > nodes: { > o > - > 4dac812b-f9df-4ecf-b0f3-e4a159d9189b: { > + name: "Sagittarius" > + transport\_address: "inet[/10.198.109.171:9300]" > } > o > - > b6b67cac-650d-46e6-b949-a6497f2d8f81: { > + name: "Crystal" > + transport\_address: "inet[/10.244.255.242:9301]" > } > o > - > e101b4b3-daa2-45fc-b042-ef14a5a56bea: { > + name: "Tinkerer" > + transport\_address: "inet[/10.244.255.242:9300]" > } > o > - > cf1b1780-3d04-4e09-b1dc-f55707acded2: { > + name: "Wolfsbane" > + transport\_address: "inet[/10.206.98.255:9300]" > } > } > - > - > > metadata: { > o > - > indices: { > + > - > index0: { > # > - > settings: { > \* index.number\_of\_replicas: "1" > \* index.number\_of\_shards: "5" > } > # > - > mappings: { > \* > - > mail: { > o > - > \_boost: { > + name: "\_boost" > } > o dynamic: true > o enabled: true > o > - > date\_formats: [ > + "dateOptionalTime" > + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" > ] > o > - > \_source: { > + enabled: false > + name: "\_source" > } > o > - > \_id: { > + store: "no" > } > o path: "full" > o > - > properties: { > + > - > to: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "to" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > body: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "body" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > receivedDateAsFloat: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: > "receivedDateAsFloat" > # index: "no" > # omit\_norms: false > # store: "yes" > # boost: 1 > # precision\_step: 4 > # term\_vector: "no" > # type: "float" > } > + > - > documentType: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "documentType" > # index: "analyzed" > # omit\_norms: false > # store: "no" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > mailId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "mailId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > subject: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "subject" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > userId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "userId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > from: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "from" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > cc: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "cc" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > receivedDate: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "receivedDate" > # index: "no" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > } > o > - > \_all: { > + enabled: true > + store: "no" > + term\_vector: "no" > } > o type: "object" > } > \* > - > calendarItem: { > o > - > \_boost: { > + name: "\_boost" > } > o dynamic: true > o enabled: true > o > - > date\_formats: [ > + "dateOptionalTime" > + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" > ] > o > - > \_source: { > + enabled: false > + name: "\_source" > } > o > - > \_id: { > + store: "no" > } > o path: "full" > o > - > properties: { > + > - > attendees: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "attendees" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > title: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "title" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > documentType: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "documentType" > # index: "analyzed" > # omit\_norms: false > # store: "no" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > details: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "details" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > location: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "location" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > start: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "start" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > userId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "userId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > organizer: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "organizer" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > calendarItemId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "calendarItemId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > } > o > - > \_all: { > + enabled: true > + store: "no" > + term\_vector: "no" > } > o type: "object" > } > \* > - > attachment: { > o > - > \_boost: { > + name: "\_boost" > } > o dynamic: true > o enabled: true > o > - > date\_formats: [ > + "dateOptionalTime" > + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" > ] > o > - > \_source: { > + enabled: false > + name: "\_source" > } > o > - > \_id: { > + store: "no" > } > o path: "full" > o > - > properties: { > + > - > attachedDate: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "attachedDate" > # index: "no" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > receivedDateAsFloat: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: > "receivedDateAsFloat" > # index: "no" > # omit\_norms: false > # store: "yes" > # boost: 1 > # precision\_step: 4 > # term\_vector: "no" > # type: "float" > } > + > - > documentType: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "documentType" > # index: "analyzed" > # omit\_norms: false > # store: "no" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > attachmentId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "attachmentId" > # index: "no" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > mailId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "mailId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > userId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "userId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > fileName: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "fileName" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > } > o > - > \_all: { > + enabled: true > + store: "no" > + term\_vector: "no" > } > o type: "object" > } > \* > - > contact: { > o > - > \_boost: { > + name: "\_boost" > } > o dynamic: true > o enabled: true > o > - > date\_formats: [ > + "dateOptionalTime" > + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd" > ] > o > - > \_source: { > + enabled: false > + name: "\_source" > } > o > - > \_id: { > + store: "no" > } > o path: "full" > o > - > properties: { > + > - > contactId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "contactId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > documentType: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "documentType" > # index: "analyzed" > # omit\_norms: false > # store: "no" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > organizations: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "organizations" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > userId: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "userId" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > fullName: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "fullName" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > emails: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "emails" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "yes" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > notes: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "notes" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > addresses: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "addresses" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > + > - > phones: { > # > omit\_term\_freq\_and\_positions: false > # index\_name: "phones" > # index: "analyzed" > # index\_analyzer: > "kihaSnowball" > # omit\_norms: false > # store: "no" > # search\_analyzer: > "kihaSnowball" > # boost: 1 > # term\_vector: "no" > # type: "string" > } > } > o > - > \_all: { > + enabled: true > + store: "no" > + term\_vector: "no" > } > o type: "object" > } > } > # aliases: > } > } > } > - > - > > routing\_table: { > o > - > indices: { > + > - > index0: { > # > - > shards: { > \* > - > 0: [ > o > - > { > + state: "INITIALIZING" > + primary: true > + node: > "cf1b1780-3d04-4e09-b1dc-f55707acded2" > + relocating\_node: null > + shard: 0 > + index: "index0" > } > o > - > { > + state: "UNASSIGNED" > + primary: false > + node: null > + relocating\_node: null > + shard: 0 > + index: "index0" > } > ] > \* > - > 1: [ > o > - > { > + state: "INITIALIZING" > + primary: false > + node: > "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" > + relocating\_node: null > + shard: 1 > + index: "index0" > } > o > - > { > + state: "STARTED" > + primary: true > + node: > "cf1b1780-3d04-4e09-b1dc-f55707acded2" > + relocating\_node: null > + shard: 1 > + index: "index0" > } > ] > \* > - > 2: [ > o > - > { > + state: "INITIALIZING" > + primary: true > + node: > "cf1b1780-3d04-4e09-b1dc-f55707acded2" > + relocating\_node: null > + shard: 2 > + index: "index0" > } > o > - > { > + state: "UNASSIGNED" > + primary: false > + node: null > + relocating\_node: null > + shard: 2 > + index: "index0" > } > ] > \* > - > 3: [ > o > - > { > + state: "INITIALIZING" > + primary: false > + node: > "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" > + relocating\_node: null > + shard: 3 > + index: "index0" > } > o > - > { > + state: "STARTED" > + primary: true > + node: > "cf1b1780-3d04-4e09-b1dc-f55707acded2" > + relocating\_node: null > + shard: 3 > + index: "index0" > } > ] > \* > - > 4: [ > o > - > { > + state: "INITIALIZING" > + primary: true > + node: > "cf1b1780-3d04-4e09-b1dc-f55707acded2" > + relocating\_node: null > + shard: 4 > + index: "index0" > } > o > - > { > + state: "UNASSIGNED" > + primary: false > + node: null > + relocating\_node: null > + shard: 4 > + index: "index0" > } > ] > } > } > } > } > - > - > > routing\_nodes: { > o > - > unassigned: [ > + > - > { > # state: "UNASSIGNED" > # primary: false > # node: null > # relocating\_node: null > # shard: 0 > # index: "index0" > } > + > - > { > # state: "UNASSIGNED" > # primary: false > # node: null > # relocating\_node: null > # shard: 2 > # index: "index0" > } > + > - > { > # state: "UNASSIGNED" > # primary: false > # node: null > # relocating\_node: null > # shard: 4 > # index: "index0" > } > ] > o > - > nodes: { > + > - > 4dac812b-f9df-4ecf-b0f3-e4a159d9189b: [ > # > - > { > \* state: "INITIALIZING" > \* primary: false > \* node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" > \* relocating\_node: null > \* shard: 1 > \* index: "index0" > } > # > - > { > \* state: "INITIALIZING" > \* primary: false > \* node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b" > \* relocating\_node: null > \* shard: 3 > \* index: "index0" > } > ] > + > - > cf1b1780-3d04-4e09-b1dc-f55707acded2: [ > # > - > { > \* state: "INITIALIZING" > \* primary: true > \* node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" > \* relocating\_node: null > \* shard: 0 > \* index: "index0" > } > # > - > { > \* state: "STARTED" > \* primary: true > \* node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" > \* relocating\_node: null > \* shard: 1 > \* index: "index0" > } > # > - > { > \* state: "INITIALIZING" > \* primary: true > \* node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" > \* relocating\_node: null > \* shard: 2 > \* index: "index0" > } > # > - > { > \* state: "STARTED" > \* primary: true > \* node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" > \* relocating\_node: null > \* shard: 3 > \* index: "index0" > } > # > - > { > \* state: "INITIALIZING" > \* primary: true > \* node: "cf1b1780-3d04-4e09-b1dc-f55707acded2" > \* relocating\_node: null > \* shard: 4 > \* index: "index0" > } > ] > } > } > > ## } > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651093.html](http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651093.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [October 7, 2010, 9:49pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/7 "2010-10-07T21:49:41Z")
Thanks for the info - will Gist next time. So if it is just recovering from shared storage, you mean it is just taking a while to download the shared indices from S3? Also, would this explain the org.elasticsearch.action.PrimaryNotStartedActionException's I see in the non-data client nodes that are trying to insert and query? Ie. state INITIALIZED is not yet STARTED? Thanks again for your help. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 7, 2010, 10:00pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/8 "2010-10-07T22:00:45Z")
Yea, it might just be recovering from s3 and it just takes time (for some reason it did not reuse the local work dir, which there were problems with pre 0.10). The indices status API has been improved in 0.10 / 0.11 to provide more information to the recovery progress. On Thu, Oct 7, 2010 at 11:49 PM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > Thanks for the info - will Gist next time. > > So if it is just recovering from shared storage, you mean it is just taking > a while to download the shared indices from S3? > > Also, would this explain the > org.elasticsearch.action.PrimaryNotStartedActionException's I see in the > non-data client nodes that are trying to insert and query? Ie. state > INITIALIZED is not yet STARTED? > > ## Thanks again for your help. > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651127.html](http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651127.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![John\_Chang](https://avatars.discourse-cdn.com/v4/letter/j/ecc23a/32.png) [@John\_Chang](https://discuss.elastic.co/u/John_Chang)\ **Post date:** [October 8, 2010, 12:37am UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/9 "2010-10-08T00:37:25Z")
I think we have hit the S3-related recovery bugs in 0.9 you mentioned above. --All the nodes that were in INITIALIZING state hours ago are still INITIALIZING. --The disk usage on the work dir of the cluster machines is not growing (it's been the same for hours), so I don't think anything more is being downloaded. Our total index volume is only 3.5 gigs. --One of the cluster machines has been logging this over and over for hours: [00:24:05,885][DEBUG][index.shard.service] [Scarlet Beetle] [index0][3] state: [CREATED]-\>[RECOVERING] [00:24:05,886][DEBUG][index.shard.service] [Scarlet Beetle] [index0][3] state: [RECOVERING]-\>[CREATED], restored after recovery So, I am actively working on upgrading to 0.11, but in the mean time, is there a work-around to get the cluster up and responsive? Thanks. ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 8, 2010, 9:27am UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/10 "2010-10-08T09:27:24Z")
You can try and reindex the data on the 0.9 one. On Fri, Oct 8, 2010 at 2:37 AM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com) wrote: > I think we have hit the S3-related recovery bugs in 0.9 you mentioned > above. > > --All the nodes that were in INITIALIZING state hours ago are still > INITIALIZING. > > --The disk usage on the work dir of the cluster machines is not growing > (it's been the same for hours), so I don't think anything more is being > downloaded. Our total index volume is only 3.5 gigs. > > --One of the cluster machines has been logging this over and over for > hours: > [00:24:05,885][DEBUG][index.shard.service] [Scarlet Beetle] > [index0][3] state: [CREATED]-\>[RECOVERING] > [00:24:05,886][DEBUG][index.shard.service] [Scarlet Beetle] > [index0][3] state: [RECOVERING]-\>[CREATED], restored after recovery > > ## So, I am actively working on upgrading to 0.11, but in the mean time, is there a work-around to get the cluster up and responsive? Thanks. > > View this message in context: > [http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html](http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html) > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Pablo\_Borges](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_borges/32/3313_2.png) [@Pablo\_Borges](https://discuss.elastic.co/u/Pablo_Borges)\ **Post date:** [October 8, 2010, 6:59pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/11 "2010-10-08T18:59:02Z")
I have exactly the same problem with Local Storage and 0.11 version os ES. The only way to recover is to reindex the data. The problem is, when the index is on that state, you cannot DELETE it, so I have to reindex all indexes. 😃 I also found out this problem occurs frequently on big indexes (big as in number of documents). Anything higher than 25k items is subject to this problem. I've removed the sharding (shard: 1) of all indexes, but the problem remains, so I have to throttle the writing and hope for the best, but I'm still seeing a crash everyday (sometimes twice a day). On Fri, Oct 8, 2010 at 6:27 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > You can try and reindex the data on the 0.9 one. > > On Fri, Oct 8, 2010 at 2:37 AM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > > > I think we have hit the S3-related recovery bugs in 0.9 you mentioned > > above. > > > > --All the nodes that were in INITIALIZING state hours ago are still > > INITIALIZING. > > > > --The disk usage on the work dir of the cluster machines is not growing > > (it's been the same for hours), so I don't think anything more is being > > downloaded. Our total index volume is only 3.5 gigs. > > > > --One of the cluster machines has been logging this over and over for > > hours: > > [00:24:05,885][DEBUG][index.shard.service] [Scarlet Beetle] > > [index0][3] state: [CREATED]-\>[RECOVERING] > > [00:24:05,886][DEBUG][index.shard.service] [Scarlet Beetle] > > [index0][3] state: [RECOVERING]-\>[CREATED], restored after recovery > > > > ## So, I am actively working on upgrading to 0.11, but in the mean time, is there a work-around to get the cluster up and responsive? Thanks. > > > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html](http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html) > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [October 8, 2010, 7:16pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/12 "2010-10-08T19:16:16Z")
First, it does not sound like exactly the same problem. And the 25k number is very bogus, don't know where you came up with it. Its like saying, it seems like once I go out with a hat, it seems to rain, hence, me wearing a hat means its going to rain. Please open another thread on the problems you have. Most times, a node will break when it runs out of memory due to capacity problems. When you open that thread, point at any exceptions you get when the node fails. -shay.banon On Fri, Oct 8, 2010 at 8:59 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > I have exactly the same problem with Local Storage and 0.11 version os ES. > > The only way to recover is to reindex the data. > > The problem is, when the index is on that state, you cannot DELETE it, so I > have to reindex all indexes. 😃 > > I also found out this problem occurs frequently on big indexes (big as in > number of documents). Anything higher than 25k items is subject to this > problem. > > I've removed the sharding (shard: 1) of all indexes, but the problem > remains, so I have to throttle the writing and hope for the best, but I'm > still seeing a crash everyday (sometimes twice a day). > > On Fri, Oct 8, 2010 at 6:27 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > You can try and reindex the data on the 0.9 one. > > > > On Fri, Oct 8, 2010 at 2:37 AM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > > > > > I think we have hit the S3-related recovery bugs in 0.9 you mentioned > > > above. > > > > > > --All the nodes that were in INITIALIZING state hours ago are still > > > INITIALIZING. > > > > > > --The disk usage on the work dir of the cluster machines is not growing > > > (it's been the same for hours), so I don't think anything more is being > > > downloaded. Our total index volume is only 3.5 gigs. > > > > > > --One of the cluster machines has been logging this over and over for > > > hours: > > > [00:24:05,885][DEBUG][index.shard.service] [Scarlet Beetle] > > > [index0][3] state: [CREATED]-\>[RECOVERING] > > > [00:24:05,886][DEBUG][index.shard.service] [Scarlet Beetle] > > > [index0][3] state: [RECOVERING]-\>[CREATED], restored after recovery > > > > > > ## So, I am actively working on upgrading to 0.11, but in the mean time, is there a work-around to get the cluster up and responsive? Thanks. > > > > > > View this message in context: > > > [http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html](http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html) > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![Pablo\_Borges](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pablo_borges/32/3313_2.png) [@Pablo\_Borges](https://discuss.elastic.co/u/Pablo_Borges)\ **Post date:** [October 8, 2010, 8:08pm UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/13 "2010-10-08T20:08:14Z")
Will do! Thanks! On Fri, Oct 8, 2010 at 4:16 PM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > First, it does not sound like exactly the same problem. And the 25k number > is very bogus, don't know where you came up with it. Its like saying, it > seems like once I go out with a hat, it seems to rain, hence, me wearing a > hat means its going to rain. > > Please open another thread on the problems you have. Most times, a node > will break when it runs out of memory due to capacity problems. When you > open that thread, point at any exceptions you get when the node fails. > > -shay.banon > > On Fri, Oct 8, 2010 at 8:59 PM, Pablo Borges [pablort@gmail.com](mailto:pablort@gmail.com) wrote: > > > I have exactly the same problem with Local Storage and 0.11 version os ES. > > > > The only way to recover is to reindex the data. > > > > The problem is, when the index is on that state, you cannot DELETE it, so > > I have to reindex all indexes. 😃 > > > > I also found out this problem occurs frequently on big indexes (big as in > > number of documents). Anything higher than 25k items is subject to this > > problem. > > > > I've removed the sharding (shard: 1) of all indexes, but the problem > > remains, so I have to throttle the writing and hope for the best, but I'm > > still seeing a crash everyday (sometimes twice a day). > > > > On Fri, Oct 8, 2010 at 6:27 AM, Shay Banon [shay.banon@elasticsearch.com](mailto:shay.banon@elasticsearch.com)wrote: > > > > > You can try and reindex the data on the 0.9 one. > > > > > > On Fri, Oct 8, 2010 at 2:37 AM, John Chang [jchangkihtest2@gmail.com](mailto:jchangkihtest2@gmail.com)wrote: > > > > > > > I think we have hit the S3-related recovery bugs in 0.9 you mentioned > > > > above. > > > > > > > > --All the nodes that were in INITIALIZING state hours ago are still > > > > INITIALIZING. > > > > > > > > --The disk usage on the work dir of the cluster machines is not growing > > > > (it's been the same for hours), so I don't think anything more is being > > > > downloaded. Our total index volume is only 3.5 gigs. > > > > > > > > --One of the cluster machines has been logging this over and over for > > > > hours: > > > > [00:24:05,885][DEBUG][index.shard.service] [Scarlet Beetle] > > > > [index0][3] state: [CREATED]-\>[RECOVERING] > > > > [00:24:05,886][DEBUG][index.shard.service] [Scarlet Beetle] > > > > [index0][3] state: [RECOVERING]-\>[CREATED], restored after recovery > > > > > > > > ## So, I am actively working on upgrading to 0.11, but in the mean time, is there a work-around to get the cluster up and responsive? Thanks. > > > > > > > > View this message in context: > > > > [http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html](http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html) > > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:18am UTC](https://discuss.elastic.co/t/broadcastshardoperationfailedexception/3422/14 "2017-07-06T04:18:11Z")