[org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to executefailure in bulk execution: UnavailableShardsException

I am doing a bulk import of all my data from mongodb into elasticsearch
using the river mongodb plugin.

I have not changed the default settings (config/elasticsearch.yml):

index.number_of_shards: 5

index.number_of_replicas: 1

I am using a single ES node.

I am using the following versions:
mongodb: 2.4.4
elasticsearch: 0.90.1
river mongodb plugin: 1.6.9

I have a script that basically does the following:

  • Backup (mongodump) of my mongodb database
  • For each collection in the database i do the following:
    • Drop the collection in mongodb
    • Create a unique river for the collection and tie it to an index
      (book-index)
    • Restore (mongorestore) the collection from the backup back into
      mongodb

From looking at the logs I can see elasticsearch indexing the new items and
i can search on the data it is indexing but i keep on getting this error in
the logs:

[2013-07-02 03:52:19,415][WARN
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to
executefailure in bulk execution:
[0]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b198],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[6]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b19e],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[11]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a3],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[16]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[21]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[25]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b5],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[31]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1bb],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[37]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1c2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[43]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1c8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[48]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ce],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[50]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1d0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[58]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1db],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[65]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1e2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[71]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ed],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[77]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f4],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[80]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f9],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[85]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ff],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[88]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b202],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[93]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b207],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[99]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b20d],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[2013-07-02 03:52:19,416][INFO
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] Indexed 100
documents, 100 insertions 0, updates, 0 deletions, 1 documents per second

How can i configure ES to remove this issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I have upped the ulimit for the elasticsearch user to 32000, it seems thats
what was causing the shards to fail which in turn caused the
UnavailableShardsException.

On Tuesday, July 2, 2013 11:01:25 PM UTC+10, Wesley Archbell wrote:

I am doing a bulk import of all my data from mongodb into elasticsearch
using the river mongodb plugin.

I have not changed the default settings (config/elasticsearch.yml):

index.number_of_shards: 5

index.number_of_replicas: 1

I am using a single ES node.

I am using the following versions:
mongodb: 2.4.4
elasticsearch: 0.90.1
river mongodb plugin: 1.6.9

I have a script that basically does the following:

  • Backup (mongodump) of my mongodb database
  • For each collection in the database i do the following:
    • Drop the collection in mongodb
    • Create a unique river for the collection and tie it to an index
      (book-index)
    • Restore (mongorestore) the collection from the backup back into
      mongodb

From looking at the logs I can see elasticsearch indexing the new items
and i can search on the data it is indexing but i keep on getting this
error in the logs:

[2013-07-02 03:52:19,415][WARN
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to
executefailure in bulk execution:
[0]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b198],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[6]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b19e],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[11]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a3],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[16]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[21]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[25]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b5],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[31]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1bb],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[37]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1c2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[43]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1c8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[48]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ce],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[50]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1d0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[58]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1db],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[65]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1e2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[71]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ed],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[77]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f4],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[80]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f9],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[85]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ff],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[88]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b202],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[93]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b207],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[99]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b20d],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[2013-07-02 03:52:19,416][INFO
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] Indexed 100
documents, 100 insertions 0, updates, 0 deletions, 1 documents per second

How can i configure ES to remove this issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.