BroadcastShardOperationFailedException

[John_Chang]

I am getting a BroadcastShardOperationFailedException when I do a GET against my cluster of two:

{

* count: 0
*
  -
  _shards: {
      o total: 5
      o successful: 3
      o failed: 2
      o
        -
        failures: [
            +
              -
              {
                  # index: "index0"
                  # shard: 0
                  # reason: "BroadcastShardOperationFailedException[[index0][0] No active shard(s)]"
              }
            +
              -
              {
                  # index: "index0"
                  # shard: 2
                  # reason: "BroadcastShardOperationFailedException[[index0][2] No active shard(s)]"
              }
        ]
  }

}

There should be 5 shards; I've restarted a number of times, and sometimes I get all 5 shards failing in essentially the same JSON as below (except it lists all shards).

In one node log I see:

[20:10:59,177][INFO ][cluster.metadata ] [Keith Kilham] [index0] creating index, cause [gateway], shards [5]/[1], mappings [mail, calendarItem, attachment, contact]
[20:11:00,005][INFO ][cluster.service ] [Keith Kilham] added {[Tinkerer][e101b4b3-daa2-45fc-b042-ef14a5a56bea][inet[/10.244.255.242:9300]]{client=true, data=false, zen.master=false},}, reason: zen-disco-receive(from node[[Tinkerer][e101b4b3-daa2-45fc-b042-ef14a5a56bea][inet[/10.244.255.242:9300]]{client=true, data=false, zen.master=false}])
[20:11:00,039][INFO ][cluster.service ] [Keith Kilham] added {[Crystal][b6b67cac-650d-46e6-b949-a6497f2d8f81][inet[/10.244.255.242:9301]]{client=true, data=false, zen.master=false},}, reason: zen-disco-receive(from node[[Crystal][b6b67cac-650d-46e6-b949-a6497f2d8f81][inet[/10.244.255.242:9301]]{client=true, data=false, zen.master=false}])
[20:11:27,547][INFO ][cluster.service ] [Keith Kilham] added {[Madelyne Pryor][c3e71382-a3c6-4c7e-bc9c-283fa9fff23a][inet[/10.198.109.171:9300]],}, reason: zen-disco-receive(from node[[Madelyne Pryor][c3e71382-a3c6-4c7e-bc9c-283fa9fff23a][inet[/10.198.109.171:9300]]])
[20:11:27,588][INFO ][http ] [Keith Kilham] bound_address {inet[/0:0:0:0:0:0:0:0:9200]}, publish_address {inet[/10.206.98.255:9200]}
[20:11:27,732][INFO ][jmx ] [Keith Kilham] bound_address {service:jmx:rmi:///jndi/rmi://:9400/jmxrmi}, publish_address {service:jmx:rmi:///jndi/rmi://10.206.98.255:9400/jmxrmi}
[20:11:27,733][INFO ][node ] [Keith Kilham] {elasticsearch/0.9.0}[11486]: started
[20:11:31,139][WARN ][index.gateway.s3 ] [Keith Kilham] [index0][0] no file [_6vm.cfs] to recover, even though it has md5, ignoring it
[20:11:31,140][WARN ][index.gateway.s3 ] [Keith Kilham] [index0][0] no file [_9bz.cfs] to recover, even though it has md5, ignoring it

In the other node log I see scrolling by over an over constantly:

[20:17:26,004][DEBUG][index.shard.service ] [Madelyne Pryor] [index0][1] state: [RECOVERING]->[CREATED], restored after recovery
[20:17:26,047][DEBUG][index.shard.service ] [Madelyne Pryor] [index0][4] state: [CREATED]->[RECOVERING]
[20:17:26,048][DEBUG][index.shard.service ] [Madelyne Pryor] [index0][4] state: [RECOVERING]->[CREATED], restored after recovery
[20:17:26,061][DEBUG][index.shard.service ] [Madelyne Pryor] [index0][3] state: [CREATED]->[RECOVERING]
[20:17:26,062][DEBUG][index.shard.service ] [Madelyne Pryor] [index0][3] state: [RECOVERING]->[CREATED], restored after recovery
[20:17:26,106][DEBUG][index.shard.service ] [Madelyne Pryor] [index0][1] state: [CREATED]->[RECOVERING]
[20:17:26,107][DEBUG][index.shard.service ] [Madelyne Pryor] [index0][1] state: [RECOVERING]->[CREATED], restored after recovery

When I write above of a 'cluster of two' I mean there are two data-carrying nodes. The third node, which is a no-data client only using the Java API, gets a PrimaryNotStartedActionException.

I am on Elastic Search 0.9. It all used to work and started failing today. Yesterday I did call the REST optimize command and I saw these problems today after a restart (the first restart since the optimize). This is deployed in a shared environment and I can't upgrade to the latest Elastic Search right away, although I plan to when I can.

Thanks for any help you can provide.

[John_Chang]

What it looks like when all the shards fail after a restart:

{

* count: 0
*
  -
  _shards: {
      o total: 5
      o successful: 0
      o failed: 5
      o
        -
        failures: [
            +
              -
              {
                  # index: "index0"
                  # shard: 0
                  # reason: "BroadcastShardOperationFailedException[[index0][0] No active shard(s)]"
              }
            +
              -
              {
                  # index: "index0"
                  # shard: 2
                  # reason: "BroadcastShardOperationFailedException[[index0][2] No active shard(s)]"
              }
            +
              -
              {
                  # index: "index0"
                  # shard: 1
                  # reason: "BroadcastShardOperationFailedException[[index0][1] No active shard(s)]"
              }
            +
              -
              {
                  # index: "index0"
                  # shard: 4
                  # reason: "BroadcastShardOperationFailedException[[index0][4] No active shard(s)]"
              }
            +
              -
              {
                  # index: "index0"
                  # shard: 3
                  # reason: "BroadcastShardOperationFailedException[[index0][3] No active shard(s)]"
              }
        ]
  }

}

[John_Chang]

More info in case it is useful.

Calling _status on one of the document types yields a NoShardAvailableActionException:

http://204.236.195.66:9200/index0/mail/_status

{

* error: "NoShardAvailableActionException[[index0][4] No shard available for [mail#_status]]"

}

However, calling _status on the index, it does know about at least some of the shards (there should be more, though, and there should be more documents, I'd expect -- I have 5 shards w/ a rep factor of 2, and it can only find 2):

http://204.236.195.66:9200/index0/_status

{

* ok: true
*
  -
  _shards: {
      o total: 10
      o successful: 2
      o failed: 0
  }
*
  -
  indices: {
      o
        -
        index0: {
            + aliases: [ ]
            +
              -
              settings: {
                  # index.number_of_replicas: "1"
                  # index.number_of_shards: "5"
              }
            + store_size: "975.4mb"
            + store_size_in_bytes: 1022851097
            + estimated_flushable_memory_size: "0b"
            + estimated_flushable_memory_size_in_bytes: 0
            + translog_operations: 65
            +
              -
              docs: {
                  # num_docs: 640340
                  # max_doc: 647256
                  # deleted_docs: 6916
              }
            +
              -
              shards: {
                  #
                    -
                    1: [
                        *
                          -
                          {
                              o
                                -
                                routing: {
                                    + state: "STARTED"
                                    + primary: true
                                    + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                                    + relocating_node: null
                                    + shard: 1
                                    + index: "index0"
                                }
                              o state: "STARTED"
                              o store_size: "489.9mb"
                              o store_size_in_bytes: 513748296
                              o estimated_flushable_memory_size: "0b"
                              o estimated_flushable_memory_size_in_bytes: 0
                              o translog_id: 1285874875878
                              o translog_operations: 33
                              o
                                -
                                docs: {
                                    + num_docs: 320026
                                    + max_doc: 323797
                                    + deleted_docs: 3771
                                }
                          }
                    ]
                  #
                    -
                    3: [
                        *
                          -
                          {
                              o
                                -
                                routing: {
                                    + state: "STARTED"
                                    + primary: true
                                    + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                                    + relocating_node: null
                                    + shard: 3
                                    + index: "index0"
                                }
                              o state: "STARTED"
                              o store_size: "485.5mb"
                              o store_size_in_bytes: 509102801
                              o estimated_flushable_memory_size: "0b"
                              o estimated_flushable_memory_size_in_bytes: 0
                              o translog_id: 1285874875925
                              o translog_operations: 32
                              o
                                -
                                docs: {
                                    + num_docs: 320314
                                    + max_doc: 323459
                                    + deleted_docs: 3145
                                }
                          }
                    ]
              }
        }
  }

}

[kimchy (Shay Banon)]

Can you run the cluster state API? Lets see where shards are allocated and
at what state they are?

On Thu, Oct 7, 2010 at 11:21 PM, John Chang jchangkihtest2@gmail.comwrote:

More info in case it is useful.

Calling _status on one of the document types yields a
NoShardAvailableActionException:

http://204.236.195.66:9200/index0/mail/_status

{

• error: "NoShardAvailableActionException[[index0][4] No shard available
  for [mail#_status]]"

}

However, calling _status on the index, it does know about at least some of
the shards (there should be more, though, and there should be more
documents, I'd expect -- I have 5 shards w/ a rep factor of 2, and it can
only find 2):

http://204.236.195.66:9200/index0/_status

{

• ok: true
• _shards: {
  o total: 10
  o successful: 2
  o failed: 0
  }
• indices: {
  o
  -
  index0: {
  + aliases:
  +
  -
  settings: {
  # index.number_of_replicas: "1"
  # index.number_of_shards: "5"
  }
  + store_size: "975.4mb"
  + store_size_in_bytes: 1022851097
  + estimated_flushable_memory_size: "0b"
  + estimated_flushable_memory_size_in_bytes: 0
  + translog_operations: 65
  +
  -
  docs: {
  # num_docs: 640340
  # max_doc: 647256
  # deleted_docs: 6916
  }
  +
  -
  shards: {
  #
  -
  1: [
  *
  -
  {
  o
  -
  routing: {
  + state: "STARTED"
  + primary: true
  + node:
  "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  + relocating_node: null
  + shard: 1
  + index: "index0"
  }
  o state: "STARTED"
  o store_size: "489.9mb"
  o store_size_in_bytes: 513748296
  o estimated_flushable_memory_size: "0b"
  o
  estimated_flushable_memory_size_in_bytes: 0
  o translog_id: 1285874875878
  o translog_operations: 33
  o
  -
  docs: {
  + num_docs: 320026
  + max_doc: 323797
  + deleted_docs: 3771
  }
  }
  ]
  #
  -
  3: [
  *
  -
  {
  o
  -
  routing: {
  + state: "STARTED"
  + primary: true
  + node:
  "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  + relocating_node: null
  + shard: 3
  + index: "index0"
  }
  o state: "STARTED"
  o store_size: "485.5mb"
  o store_size_in_bytes: 509102801
  o estimated_flushable_memory_size: "0b"
  o
  estimated_flushable_memory_size_in_bytes: 0
  o translog_id: 1285874875925
  o translog_operations: 32
  o
  -
  docs: {
  + num_docs: 320314
  + max_doc: 323459
  + deleted_docs: 3145
  }
  }
  ]
  }
  }
  }

}

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1650984.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

[John_Chang]

http://204.236.195.66:9200/_cluster/state?pretty=true

produces:

{

* cluster_name: "dev5"
* master_node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
* blocks: { }
*
  -
  nodes: {
      o
        -
        4dac812b-f9df-4ecf-b0f3-e4a159d9189b: {
            + name: "Sagittarius"
            + transport_address: "inet[/10.198.109.171:9300]"
        }
      o
        -
        b6b67cac-650d-46e6-b949-a6497f2d8f81: {
            + name: "Crystal"
            + transport_address: "inet[/10.244.255.242:9301]"
        }
      o
        -
        e101b4b3-daa2-45fc-b042-ef14a5a56bea: {
            + name: "Tinkerer"
            + transport_address: "inet[/10.244.255.242:9300]"
        }
      o
        -
        cf1b1780-3d04-4e09-b1dc-f55707acded2: {
            + name: "Wolfsbane"
            + transport_address: "inet[/10.206.98.255:9300]"
        }
  }
*
  -
  metadata: {
      o
        -
        indices: {
            +
              -
              index0: {
                  #
                    -
                    settings: {
                        * index.number_of_replicas: "1"
                        * index.number_of_shards: "5"
                    }
                  #
                    -
                    mappings: {
                        *
                          -
                          mail: {
                              o
                                -
                                _boost: {
                                    + name: "_boost"
                                }
                              o dynamic: true
                              o enabled: true
                              o
                                -
                                date_formats: [
                                    + "dateOptionalTime"
                                    + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
                                ]
                              o
                                -
                                _source: {
                                    + enabled: false
                                    + name: "_source"
                                }
                              o
                                -
                                _id: {
                                    + store: "no"
                                }
                              o path: "full"
                              o
                                -
                                properties: {
                                    +
                                      -
                                      to: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "to"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      body: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "body"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      receivedDateAsFloat: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "receivedDateAsFloat"
                                          # index: "no"
                                          # omit_norms: false
                                          # store: "yes"
                                          # boost: 1
                                          # precision_step: 4
                                          # term_vector: "no"
                                          # type: "float"
                                      }
                                    +
                                      -
                                      documentType: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "documentType"
                                          # index: "analyzed"
                                          # omit_norms: false
                                          # store: "no"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      mailId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "mailId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      subject: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "subject"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      userId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "userId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      from: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "from"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      cc: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "cc"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      receivedDate: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "receivedDate"
                                          # index: "no"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                }
                              o
                                -
                                _all: {
                                    + enabled: true
                                    + store: "no"
                                    + term_vector: "no"
                                }
                              o type: "object"
                          }
                        *
                          -
                          calendarItem: {
                              o
                                -
                                _boost: {
                                    + name: "_boost"
                                }
                              o dynamic: true
                              o enabled: true
                              o
                                -
                                date_formats: [
                                    + "dateOptionalTime"
                                    + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
                                ]
                              o
                                -
                                _source: {
                                    + enabled: false
                                    + name: "_source"
                                }
                              o
                                -
                                _id: {
                                    + store: "no"
                                }
                              o path: "full"
                              o
                                -
                                properties: {
                                    +
                                      -
                                      attendees: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "attendees"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      title: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "title"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      documentType: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "documentType"
                                          # index: "analyzed"
                                          # omit_norms: false
                                          # store: "no"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      details: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "details"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      location: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "location"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      start: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "start"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      userId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "userId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      organizer: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "organizer"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      calendarItemId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "calendarItemId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                }
                              o
                                -
                                _all: {
                                    + enabled: true
                                    + store: "no"
                                    + term_vector: "no"
                                }
                              o type: "object"
                          }
                        *
                          -
                          attachment: {
                              o
                                -
                                _boost: {
                                    + name: "_boost"
                                }
                              o dynamic: true
                              o enabled: true
                              o
                                -
                                date_formats: [
                                    + "dateOptionalTime"
                                    + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
                                ]
                              o
                                -
                                _source: {
                                    + enabled: false
                                    + name: "_source"
                                }
                              o
                                -
                                _id: {
                                    + store: "no"
                                }
                              o path: "full"
                              o
                                -
                                properties: {
                                    +
                                      -
                                      attachedDate: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "attachedDate"
                                          # index: "no"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      receivedDateAsFloat: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "receivedDateAsFloat"
                                          # index: "no"
                                          # omit_norms: false
                                          # store: "yes"
                                          # boost: 1
                                          # precision_step: 4
                                          # term_vector: "no"
                                          # type: "float"
                                      }
                                    +
                                      -
                                      documentType: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "documentType"
                                          # index: "analyzed"
                                          # omit_norms: false
                                          # store: "no"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      attachmentId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "attachmentId"
                                          # index: "no"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      mailId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "mailId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      userId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "userId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      fileName: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "fileName"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                }
                              o
                                -
                                _all: {
                                    + enabled: true
                                    + store: "no"
                                    + term_vector: "no"
                                }
                              o type: "object"
                          }
                        *
                          -
                          contact: {
                              o
                                -
                                _boost: {
                                    + name: "_boost"
                                }
                              o dynamic: true
                              o enabled: true
                              o
                                -
                                date_formats: [
                                    + "dateOptionalTime"
                                    + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
                                ]
                              o
                                -
                                _source: {
                                    + enabled: false
                                    + name: "_source"
                                }
                              o
                                -
                                _id: {
                                    + store: "no"
                                }
                              o path: "full"
                              o
                                -
                                properties: {
                                    +
                                      -
                                      contactId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "contactId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      documentType: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "documentType"
                                          # index: "analyzed"
                                          # omit_norms: false
                                          # store: "no"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      organizations: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "organizations"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      userId: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "userId"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      fullName: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "fullName"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      emails: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "emails"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "yes"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      notes: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "notes"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      addresses: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "addresses"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                    +
                                      -
                                      phones: {
                                          # omit_term_freq_and_positions: false
                                          # index_name: "phones"
                                          # index: "analyzed"
                                          # index_analyzer: "kihaSnowball"
                                          # omit_norms: false
                                          # store: "no"
                                          # search_analyzer: "kihaSnowball"
                                          # boost: 1
                                          # term_vector: "no"
                                          # type: "string"
                                      }
                                }
                              o
                                -
                                _all: {
                                    + enabled: true
                                    + store: "no"
                                    + term_vector: "no"
                                }
                              o type: "object"
                          }
                    }
                  # aliases: [ ]
              }
        }
  }
*
  -
  routing_table: {
      o
        -
        indices: {
            +
              -
              index0: {
                  #
                    -
                    shards: {
                        *
                          -
                          0: [
                              o
                                -
                                {
                                    + state: "INITIALIZING"
                                    + primary: true
                                    + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                                    + relocating_node: null
                                    + shard: 0
                                    + index: "index0"
                                }
                              o
                                -
                                {
                                    + state: "UNASSIGNED"
                                    + primary: false
                                    + node: null
                                    + relocating_node: null
                                    + shard: 0
                                    + index: "index0"
                                }
                          ]
                        *
                          -
                          1: [
                              o
                                -
                                {
                                    + state: "INITIALIZING"
                                    + primary: false
                                    + node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
                                    + relocating_node: null
                                    + shard: 1
                                    + index: "index0"
                                }
                              o
                                -
                                {
                                    + state: "STARTED"
                                    + primary: true
                                    + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                                    + relocating_node: null
                                    + shard: 1
                                    + index: "index0"
                                }
                          ]
                        *
                          -
                          2: [
                              o
                                -
                                {
                                    + state: "INITIALIZING"
                                    + primary: true
                                    + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                                    + relocating_node: null
                                    + shard: 2
                                    + index: "index0"
                                }
                              o
                                -
                                {
                                    + state: "UNASSIGNED"
                                    + primary: false
                                    + node: null
                                    + relocating_node: null
                                    + shard: 2
                                    + index: "index0"
                                }
                          ]
                        *
                          -
                          3: [
                              o
                                -
                                {
                                    + state: "INITIALIZING"
                                    + primary: false
                                    + node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
                                    + relocating_node: null
                                    + shard: 3
                                    + index: "index0"
                                }
                              o
                                -
                                {
                                    + state: "STARTED"
                                    + primary: true
                                    + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                                    + relocating_node: null
                                    + shard: 3
                                    + index: "index0"
                                }
                          ]
                        *
                          -
                          4: [
                              o
                                -
                                {
                                    + state: "INITIALIZING"
                                    + primary: true
                                    + node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                                    + relocating_node: null
                                    + shard: 4
                                    + index: "index0"
                                }
                              o
                                -
                                {
                                    + state: "UNASSIGNED"
                                    + primary: false
                                    + node: null
                                    + relocating_node: null
                                    + shard: 4
                                    + index: "index0"
                                }
                          ]
                    }
              }
        }
  }
*
  -
  routing_nodes: {
      o
        -
        unassigned: [
            +
              -
              {
                  # state: "UNASSIGNED"
                  # primary: false
                  # node: null
                  # relocating_node: null
                  # shard: 0
                  # index: "index0"
              }
            +
              -
              {
                  # state: "UNASSIGNED"
                  # primary: false
                  # node: null
                  # relocating_node: null
                  # shard: 2
                  # index: "index0"
              }
            +
              -
              {
                  # state: "UNASSIGNED"
                  # primary: false
                  # node: null
                  # relocating_node: null
                  # shard: 4
                  # index: "index0"
              }
        ]
      o
        -
        nodes: {
            +
              -
              4dac812b-f9df-4ecf-b0f3-e4a159d9189b: [
                  #
                    -
                    {
                        * state: "INITIALIZING"
                        * primary: false
                        * node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
                        * relocating_node: null
                        * shard: 1
                        * index: "index0"
                    }
                  #
                    -
                    {
                        * state: "INITIALIZING"
                        * primary: false
                        * node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
                        * relocating_node: null
                        * shard: 3
                        * index: "index0"
                    }
              ]
            +
              -
              cf1b1780-3d04-4e09-b1dc-f55707acded2: [
                  #
                    -
                    {
                        * state: "INITIALIZING"
                        * primary: true
                        * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                        * relocating_node: null
                        * shard: 0
                        * index: "index0"
                    }
                  #
                    -
                    {
                        * state: "STARTED"
                        * primary: true
                        * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                        * relocating_node: null
                        * shard: 1
                        * index: "index0"
                    }
                  #
                    -
                    {
                        * state: "INITIALIZING"
                        * primary: true
                        * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                        * relocating_node: null
                        * shard: 2
                        * index: "index0"
                    }
                  #
                    -
                    {
                        * state: "STARTED"
                        * primary: true
                        * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                        * relocating_node: null
                        * shard: 3
                        * index: "index0"
                    }
                  #
                    -
                    {
                        * state: "INITIALIZING"
                        * primary: true
                        * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
                        * relocating_node: null
                        * shard: 4
                        * index: "index0"
                    }
              ]
        }
  }

}

[kimchy (Shay Banon)]

Seems like the shards are in initializing state, maybe it just takes time to
recover from the shared storage? If you don't see exceptions, this is what
they do...

There were some bugs around it in 0.9, not sure if you hit one of them or
not.... .

-shay.banon

p.s. Can you gist the info next time? Hard to make sense of it in mails...

On Thu, Oct 7, 2010 at 11:42 PM, John Chang jchangkihtest2@gmail.comwrote:

http://204.236.195.66:9200/_cluster/state?pretty=true

produces:

{

• cluster_name: "dev5"
• master_node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
• blocks: { }
• nodes: {
  o
  -
  4dac812b-f9df-4ecf-b0f3-e4a159d9189b: {
  + name: "Sagittarius"
  + transport_address: "inet[/10.198.109.171:9300]"
  }
  o
  -
  b6b67cac-650d-46e6-b949-a6497f2d8f81: {
  + name: "Crystal"
  + transport_address: "inet[/10.244.255.242:9301]"
  }
  o
  -
  e101b4b3-daa2-45fc-b042-ef14a5a56bea: {
  + name: "Tinkerer"
  + transport_address: "inet[/10.244.255.242:9300]"
  }
  o
  -
  cf1b1780-3d04-4e09-b1dc-f55707acded2: {
  + name: "Wolfsbane"
  + transport_address: "inet[/10.206.98.255:9300]"
  }
  }
• metadata: {
  o
  -
  indices: {
  +
  -
  index0: {
  #
  -
  settings: {
  * index.number_of_replicas: "1"
  * index.number_of_shards: "5"
  }
  #
  -
  mappings: {
  *
  -
  mail: {
  o
  -
  _boost: {
  + name: "_boost"
  }
  o dynamic: true
  o enabled: true
  o
  -
  date_formats: [
  + "dateOptionalTime"
  + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
  ]
  o
  -
  _source: {
  + enabled: false
  + name: "_source"
  }
  o
  -
  _id: {
  + store: "no"
  }
  o path: "full"
  o
  -
  properties: {
  +
  -
  to: {
  #
  omit_term_freq_and_positions: false
  # index_name: "to"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  body: {
  #
  omit_term_freq_and_positions: false
  # index_name: "body"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  receivedDateAsFloat: {
  #
  omit_term_freq_and_positions: false
  # index_name:
  "receivedDateAsFloat"
  # index: "no"
  # omit_norms: false
  # store: "yes"
  # boost: 1
  # precision_step: 4
  # term_vector: "no"
  # type: "float"
  }
  +
  -
  documentType: {
  #
  omit_term_freq_and_positions: false
  # index_name: "documentType"
  # index: "analyzed"
  # omit_norms: false
  # store: "no"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  mailId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "mailId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  subject: {
  #
  omit_term_freq_and_positions: false
  # index_name: "subject"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  userId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "userId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  from: {
  #
  omit_term_freq_and_positions: false
  # index_name: "from"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  cc: {
  #
  omit_term_freq_and_positions: false
  # index_name: "cc"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  receivedDate: {
  #
  omit_term_freq_and_positions: false
  # index_name: "receivedDate"
  # index: "no"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  }
  o
  -
  _all: {
  + enabled: true
  + store: "no"
  + term_vector: "no"
  }
  o type: "object"
  }
  *
  -
  calendarItem: {
  o
  -
  _boost: {
  + name: "_boost"
  }
  o dynamic: true
  o enabled: true
  o
  -
  date_formats: [
  + "dateOptionalTime"
  + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
  ]
  o
  -
  _source: {
  + enabled: false
  + name: "_source"
  }
  o
  -
  _id: {
  + store: "no"
  }
  o path: "full"
  o
  -
  properties: {
  +
  -
  attendees: {
  #
  omit_term_freq_and_positions: false
  # index_name: "attendees"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  title: {
  #
  omit_term_freq_and_positions: false
  # index_name: "title"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  documentType: {
  #
  omit_term_freq_and_positions: false
  # index_name: "documentType"
  # index: "analyzed"
  # omit_norms: false
  # store: "no"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  details: {
  #
  omit_term_freq_and_positions: false
  # index_name: "details"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  location: {
  #
  omit_term_freq_and_positions: false
  # index_name: "location"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  start: {
  #
  omit_term_freq_and_positions: false
  # index_name: "start"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  userId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "userId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  organizer: {
  #
  omit_term_freq_and_positions: false
  # index_name: "organizer"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  calendarItemId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "calendarItemId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  }
  o
  -
  _all: {
  + enabled: true
  + store: "no"
  + term_vector: "no"
  }
  o type: "object"
  }
  *
  -
  attachment: {
  o
  -
  _boost: {
  + name: "_boost"
  }
  o dynamic: true
  o enabled: true
  o
  -
  date_formats: [
  + "dateOptionalTime"
  + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
  ]
  o
  -
  _source: {
  + enabled: false
  + name: "_source"
  }
  o
  -
  _id: {
  + store: "no"
  }
  o path: "full"
  o
  -
  properties: {
  +
  -
  attachedDate: {
  #
  omit_term_freq_and_positions: false
  # index_name: "attachedDate"
  # index: "no"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  receivedDateAsFloat: {
  #
  omit_term_freq_and_positions: false
  # index_name:
  "receivedDateAsFloat"
  # index: "no"
  # omit_norms: false
  # store: "yes"
  # boost: 1
  # precision_step: 4
  # term_vector: "no"
  # type: "float"
  }
  +
  -
  documentType: {
  #
  omit_term_freq_and_positions: false
  # index_name: "documentType"
  # index: "analyzed"
  # omit_norms: false
  # store: "no"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  attachmentId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "attachmentId"
  # index: "no"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  mailId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "mailId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  userId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "userId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  fileName: {
  #
  omit_term_freq_and_positions: false
  # index_name: "fileName"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  }
  o
  -
  _all: {
  + enabled: true
  + store: "no"
  + term_vector: "no"
  }
  o type: "object"
  }
  *
  -
  contact: {
  o
  -
  _boost: {
  + name: "_boost"
  }
  o dynamic: true
  o enabled: true
  o
  -
  date_formats: [
  + "dateOptionalTime"
  + "yyyy/MM/dd HH:mm:ss||yyyy/MM/dd"
  ]
  o
  -
  _source: {
  + enabled: false
  + name: "_source"
  }
  o
  -
  _id: {
  + store: "no"
  }
  o path: "full"
  o
  -
  properties: {
  +
  -
  contactId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "contactId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  documentType: {
  #
  omit_term_freq_and_positions: false
  # index_name: "documentType"
  # index: "analyzed"
  # omit_norms: false
  # store: "no"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  organizations: {
  #
  omit_term_freq_and_positions: false
  # index_name: "organizations"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  userId: {
  #
  omit_term_freq_and_positions: false
  # index_name: "userId"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  fullName: {
  #
  omit_term_freq_and_positions: false
  # index_name: "fullName"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  emails: {
  #
  omit_term_freq_and_positions: false
  # index_name: "emails"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "yes"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  notes: {
  #
  omit_term_freq_and_positions: false
  # index_name: "notes"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  addresses: {
  #
  omit_term_freq_and_positions: false
  # index_name: "addresses"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  +
  -
  phones: {
  #
  omit_term_freq_and_positions: false
  # index_name: "phones"
  # index: "analyzed"
  # index_analyzer:
  "kihaSnowball"
  # omit_norms: false
  # store: "no"
  # search_analyzer:
  "kihaSnowball"
  # boost: 1
  # term_vector: "no"
  # type: "string"
  }
  }
  o
  -
  _all: {
  + enabled: true
  + store: "no"
  + term_vector: "no"
  }
  o type: "object"
  }
  }
  # aliases:
  }
  }
  }
• routing_table: {
  o
  -
  indices: {
  +
  -
  index0: {
  #
  -
  shards: {
  *
  -
  0: [
  o
  -
  {
  + state: "INITIALIZING"
  + primary: true
  + node:
  "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  + relocating_node: null
  + shard: 0
  + index: "index0"
  }
  o
  -
  {
  + state: "UNASSIGNED"
  + primary: false
  + node: null
  + relocating_node: null
  + shard: 0
  + index: "index0"
  }
  ]
  *
  -
  1: [
  o
  -
  {
  + state: "INITIALIZING"
  + primary: false
  + node:
  "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
  + relocating_node: null
  + shard: 1
  + index: "index0"
  }
  o
  -
  {
  + state: "STARTED"
  + primary: true
  + node:
  "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  + relocating_node: null
  + shard: 1
  + index: "index0"
  }
  ]
  *
  -
  2: [
  o
  -
  {
  + state: "INITIALIZING"
  + primary: true
  + node:
  "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  + relocating_node: null
  + shard: 2
  + index: "index0"
  }
  o
  -
  {
  + state: "UNASSIGNED"
  + primary: false
  + node: null
  + relocating_node: null
  + shard: 2
  + index: "index0"
  }
  ]
  *
  -
  3: [
  o
  -
  {
  + state: "INITIALIZING"
  + primary: false
  + node:
  "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
  + relocating_node: null
  + shard: 3
  + index: "index0"
  }
  o
  -
  {
  + state: "STARTED"
  + primary: true
  + node:
  "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  + relocating_node: null
  + shard: 3
  + index: "index0"
  }
  ]
  *
  -
  4: [
  o
  -
  {
  + state: "INITIALIZING"
  + primary: true
  + node:
  "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  + relocating_node: null
  + shard: 4
  + index: "index0"
  }
  o
  -
  {
  + state: "UNASSIGNED"
  + primary: false
  + node: null
  + relocating_node: null
  + shard: 4
  + index: "index0"
  }
  ]
  }
  }
  }
  }
• routing_nodes: {
  o
  -
  unassigned: [
  +
  -
  {
  # state: "UNASSIGNED"
  # primary: false
  # node: null
  # relocating_node: null
  # shard: 0
  # index: "index0"
  }
  +
  -
  {
  # state: "UNASSIGNED"
  # primary: false
  # node: null
  # relocating_node: null
  # shard: 2
  # index: "index0"
  }
  +
  -
  {
  # state: "UNASSIGNED"
  # primary: false
  # node: null
  # relocating_node: null
  # shard: 4
  # index: "index0"
  }
  ]
  o
  -
  nodes: {
  +
  -
  4dac812b-f9df-4ecf-b0f3-e4a159d9189b: [
  #
  -
  {
  * state: "INITIALIZING"
  * primary: false
  * node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
  * relocating_node: null
  * shard: 1
  * index: "index0"
  }
  #
  -
  {
  * state: "INITIALIZING"
  * primary: false
  * node: "4dac812b-f9df-4ecf-b0f3-e4a159d9189b"
  * relocating_node: null
  * shard: 3
  * index: "index0"
  }
  ]
  +
  -
  cf1b1780-3d04-4e09-b1dc-f55707acded2: [
  #
  -
  {
  * state: "INITIALIZING"
  * primary: true
  * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  * relocating_node: null
  * shard: 0
  * index: "index0"
  }
  #
  -
  {
  * state: "STARTED"
  * primary: true
  * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  * relocating_node: null
  * shard: 1
  * index: "index0"
  }
  #
  -
  {
  * state: "INITIALIZING"
  * primary: true
  * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  * relocating_node: null
  * shard: 2
  * index: "index0"
  }
  #
  -
  {
  * state: "STARTED"
  * primary: true
  * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  * relocating_node: null
  * shard: 3
  * index: "index0"
  }
  #
  -
  {
  * state: "INITIALIZING"
  * primary: true
  * node: "cf1b1780-3d04-4e09-b1dc-f55707acded2"
  * relocating_node: null
  * shard: 4
  * index: "index0"
  }
  ]
  }
  }

}

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651093.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

[John_Chang]

Thanks for the info - will Gist next time.

So if it is just recovering from shared storage, you mean it is just taking a while to download the shared indices from S3?

Also, would this explain the org.elasticsearch.action.PrimaryNotStartedActionException's I see in the non-data client nodes that are trying to insert and query? Ie. state INITIALIZED is not yet STARTED?

Thanks again for your help.

[kimchy (Shay Banon)]

Yea, it might just be recovering from s3 and it just takes time (for some
reason it did not reuse the local work dir, which there were problems with
pre 0.10). The indices status API has been improved in 0.10 / 0.11 to
provide more information to the recovery progress.

On Thu, Oct 7, 2010 at 11:49 PM, John Chang jchangkihtest2@gmail.comwrote:

Thanks for the info - will Gist next time.

So if it is just recovering from shared storage, you mean it is just taking
a while to download the shared indices from S3?

Also, would this explain the
org.elasticsearch.action.PrimaryNotStartedActionException's I see in the
non-data client nodes that are trying to insert and query? Ie. state
INITIALIZED is not yet STARTED?

Thanks again for your help.

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651127.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

[John_Chang]

I think we have hit the S3-related recovery bugs in 0.9 you mentioned above.

--All the nodes that were in INITIALIZING state hours ago are still INITIALIZING.

--The disk usage on the work dir of the cluster machines is not growing (it's been the same for hours), so I don't think anything more is being downloaded. Our total index volume is only 3.5 gigs.

--One of the cluster machines has been logging this over and over for hours:
[00:24:05,885][DEBUG][index.shard.service ] [Scarlet Beetle] [index0][3] state: [CREATED]->[RECOVERING]
[00:24:05,886][DEBUG][index.shard.service ] [Scarlet Beetle] [index0][3] state: [RECOVERING]->[CREATED], restored after recovery

So, I am actively working on upgrading to 0.11, but in the mean time, is there a work-around to get the cluster up and responsive? Thanks.

[kimchy (Shay Banon)]

You can try and reindex the data on the 0.9 one.

On Fri, Oct 8, 2010 at 2:37 AM, John Chang jchangkihtest2@gmail.com wrote:

I think we have hit the S3-related recovery bugs in 0.9 you mentioned
above.

--All the nodes that were in INITIALIZING state hours ago are still
INITIALIZING.

--The disk usage on the work dir of the cluster machines is not growing
(it's been the same for hours), so I don't think anything more is being
downloaded. Our total index volume is only 3.5 gigs.

--One of the cluster machines has been logging this over and over for
hours:
[00:24:05,885][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [CREATED]->[RECOVERING]
[00:24:05,886][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [RECOVERING]->[CREATED], restored after recovery

So, I am actively working on upgrading to 0.11, but in the mean time, is
there a work-around to get the cluster up and responsive? Thanks.

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

[Pablo_Borges]

I have exactly the same problem with Local Storage and 0.11 version os ES.

The only way to recover is to reindex the data.

The problem is, when the index is on that state, you cannot DELETE it, so I
have to reindex all indexes.

I also found out this problem occurs frequently on big indexes (big as in
number of documents). Anything higher than 25k items is subject to this
problem.

I've removed the sharding (shard: 1) of all indexes, but the problem
remains, so I have to throttle the writing and hope for the best, but I'm
still seeing a crash everyday (sometimes twice a day).

On Fri, Oct 8, 2010 at 6:27 AM, Shay Banon shay.banon@elasticsearch.comwrote:

You can try and reindex the data on the 0.9 one.

On Fri, Oct 8, 2010 at 2:37 AM, John Chang jchangkihtest2@gmail.comwrote:

I think we have hit the S3-related recovery bugs in 0.9 you mentioned
above.

--All the nodes that were in INITIALIZING state hours ago are still
INITIALIZING.

--The disk usage on the work dir of the cluster machines is not growing
(it's been the same for hours), so I don't think anything more is being
downloaded. Our total index volume is only 3.5 gigs.

--One of the cluster machines has been logging this over and over for
hours:
[00:24:05,885][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [CREATED]->[RECOVERING]
[00:24:05,886][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [RECOVERING]->[CREATED], restored after recovery

So, I am actively working on upgrading to 0.11, but in the mean time, is
there a work-around to get the cluster up and responsive? Thanks.

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

[kimchy (Shay Banon)]

First, it does not sound like exactly the same problem. And the 25k number
is very bogus, don't know where you came up with it. Its like saying, it
seems like once I go out with a hat, it seems to rain, hence, me wearing a
hat means its going to rain.

Please open another thread on the problems you have. Most times, a node will
break when it runs out of memory due to capacity problems. When you open
that thread, point at any exceptions you get when the node fails.

-shay.banon

On Fri, Oct 8, 2010 at 8:59 PM, Pablo Borges pablort@gmail.com wrote:

I have exactly the same problem with Local Storage and 0.11 version os ES.

The only way to recover is to reindex the data.

The problem is, when the index is on that state, you cannot DELETE it, so I
have to reindex all indexes.

I also found out this problem occurs frequently on big indexes (big as in
number of documents). Anything higher than 25k items is subject to this
problem.

I've removed the sharding (shard: 1) of all indexes, but the problem
remains, so I have to throttle the writing and hope for the best, but I'm
still seeing a crash everyday (sometimes twice a day).

On Fri, Oct 8, 2010 at 6:27 AM, Shay Banon shay.banon@elasticsearch.comwrote:

You can try and reindex the data on the 0.9 one.

On Fri, Oct 8, 2010 at 2:37 AM, John Chang jchangkihtest2@gmail.comwrote:

I think we have hit the S3-related recovery bugs in 0.9 you mentioned
above.

--All the nodes that were in INITIALIZING state hours ago are still
INITIALIZING.

--The disk usage on the work dir of the cluster machines is not growing
(it's been the same for hours), so I don't think anything more is being
downloaded. Our total index volume is only 3.5 gigs.

--One of the cluster machines has been logging this over and over for
hours:
[00:24:05,885][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [CREATED]->[RECOVERING]
[00:24:05,886][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [RECOVERING]->[CREATED], restored after recovery

So, I am actively working on upgrading to 0.11, but in the mean time, is
there a work-around to get the cluster up and responsive? Thanks.

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

[Pablo_Borges]

Will do!

Thanks!

On Fri, Oct 8, 2010 at 4:16 PM, Shay Banon shay.banon@elasticsearch.comwrote:

First, it does not sound like exactly the same problem. And the 25k number
is very bogus, don't know where you came up with it. Its like saying, it
seems like once I go out with a hat, it seems to rain, hence, me wearing a
hat means its going to rain.

Please open another thread on the problems you have. Most times, a node
will break when it runs out of memory due to capacity problems. When you
open that thread, point at any exceptions you get when the node fails.

-shay.banon

On Fri, Oct 8, 2010 at 8:59 PM, Pablo Borges pablort@gmail.com wrote:

I have exactly the same problem with Local Storage and 0.11 version os ES.

The only way to recover is to reindex the data.

The problem is, when the index is on that state, you cannot DELETE it, so
I have to reindex all indexes.

I also found out this problem occurs frequently on big indexes (big as in
number of documents). Anything higher than 25k items is subject to this
problem.

I've removed the sharding (shard: 1) of all indexes, but the problem
remains, so I have to throttle the writing and hope for the best, but I'm
still seeing a crash everyday (sometimes twice a day).

On Fri, Oct 8, 2010 at 6:27 AM, Shay Banon shay.banon@elasticsearch.comwrote:

You can try and reindex the data on the 0.9 one.

On Fri, Oct 8, 2010 at 2:37 AM, John Chang jchangkihtest2@gmail.comwrote:

I think we have hit the S3-related recovery bugs in 0.9 you mentioned
above.

--All the nodes that were in INITIALIZING state hours ago are still
INITIALIZING.

--The disk usage on the work dir of the cluster machines is not growing
(it's been the same for hours), so I don't think anything more is being
downloaded. Our total index volume is only 3.5 gigs.

--One of the cluster machines has been logging this over and over for
hours:
[00:24:05,885][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [CREATED]->[RECOVERING]
[00:24:05,886][DEBUG][index.shard.service ] [Scarlet Beetle]
[index0][3] state: [RECOVERING]->[CREATED], restored after recovery

So, I am actively working on upgrading to 0.11, but in the mean time, is
there a work-around to get the cluster up and responsive? Thanks.

View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/BroadcastShardOperationFailedException-tp1650661p1651779.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.