[org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to executefailure in bulk execution: UnavailableShardsException

I am doing a bulk import of all my data from mongodb into elasticsearch
using the river mongodb plugin.

I have not changed the default settings (config/elasticsearch.yml):

index.number_of_shards: 5

index.number_of_replicas: 1

I am using a single ES node.

I am using the following versions:
mongodb: 2.4.4
elasticsearch: 0.90.1
river mongodb plugin: 1.6.9

I have a script that basically does the following:

  • Backup (mongodump) of my mongodb database
  • For each collection in the database i do the following:
    • Drop the collection in mongodb
    • Create a unique river for the collection and tie it to an index
      (book-index)
    • Restore (mongorestore) the collection from the backup back into
      mongodb

From looking at the logs I can see elasticsearch indexing the new items and
i can search on the data it is indexing but i keep on getting this error in
the logs:

[2013-07-02 03:52:19,415][WARN
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to
executefailure in bulk execution:
[0]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b198],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[6]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b19e],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[11]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a3],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[16]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[21]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[25]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b5],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[31]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1bb],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[37]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1c2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[43]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1c8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[48]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ce],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[50]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1d0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[58]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1db],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[65]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1e2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[71]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ed],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[77]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f4],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[80]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f9],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[85]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ff],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[88]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b202],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[93]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b207],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[99]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b20d],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[2013-07-02 03:52:19,416][INFO
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] Indexed 100
documents, 100 insertions 0, updates, 0 deletions, 1 documents per second

How can i configure ES to remove this issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I have upped the ulimit for the elasticsearch user to 32000, it seems thats
what was causing the shards to fail which in turn caused the
UnavailableShardsException.

http://www.elasticsearch.org/tutorials/too-many-open-files/

On Tuesday, July 2, 2013 11:01:25 PM UTC+10, Wesley Archbell wrote:

I am doing a bulk import of all my data from mongodb into elasticsearch
using the river mongodb plugin.

I have not changed the default settings (config/elasticsearch.yml):

index.number_of_shards: 5

index.number_of_replicas: 1

I am using a single ES node.

I am using the following versions:
mongodb: 2.4.4
elasticsearch: 0.90.1
river mongodb plugin: 1.6.9

I have a script that basically does the following:

  • Backup (mongodump) of my mongodb database
  • For each collection in the database i do the following:
    • Drop the collection in mongodb
    • Create a unique river for the collection and tie it to an index
      (book-index)
    • Restore (mongorestore) the collection from the backup back into
      mongodb

From looking at the logs I can see elasticsearch indexing the new items
and i can search on the data it is indexing but i keep on getting this
error in the logs:

[2013-07-02 03:52:19,415][WARN
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] failed to
executefailure in bulk execution:
[0]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b198],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[6]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b19e],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[11]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a3],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[16]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1a8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[21]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[25]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1b5],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[31]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1bb],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[37]: index [book-index], type [book], id [4fbe6dc1e4b0f55a4713b1c2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[43]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1c8],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[48]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ce],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[50]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1d0],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[58]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1db],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[65]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1e2],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[71]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ed],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[77]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f4],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[80]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1f9],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[85]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b1ff],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[88]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b202],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[93]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b207],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[99]: index [book-index], type [book], id [4fbe6dc2e4b0f55a4713b20d],
message [UnavailableShardsException[[book-index][1] [2] shardIt, [0] active
: Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@19a07f6https://github.com/org.elasticsearch.action.bulk.BulkShardRequest/elasticsearch-river-mongodb/commit/19a07f6e
]]
[2013-07-02 03:52:19,416][INFO
][org.elasticsearch.river.mongodb.MongoDBRiver$Indexer] Indexed 100
documents, 100 insertions 0, updates, 0 deletions, 1 documents per second

How can i configure ES to remove this issue?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.