Problem with bulk insertion

vinod_eligeti · July 2, 2013, 10:21pm

I have an index which is of the format:

lk: where lk is a prefix and timestamp is stripped to a
month

The embedded node i created has 1 shard and no replication which i gave
something like this:

NodeBuilder nb = new NodeBuilder().clusterName("log-store");
nb.settings().put("node.data", true);
nb.settings().put("index.number_of_shards", 1);
nb.settings().put("index.number_of_replicas", 0);

If i create an index, below for example:

lk039dbfc1-95cd-42d2-8e9c-9799423466de:1372723200000

Then bulk insertion happens fine and happens quickly without much delay.
However if the create an index without the hyphen like:

lk039dbfc195cd42d28e9c9799423466de:1372723200000

The bulk insertion throws an exception after waiting 1m.

0]: index [lk0c646caeba0d4d2d84d9f3dbb0e018e3:1372723200000], type [docs],
id [PPAfs1r0QMeBr7PFH0wsBQ], message
[UnavailableShardsException[[lk0c646caeba0d4d2d84d9f3dbb0e018e3:1372723200000][0]
[1] shardIt, [0] active : Timeout waiting for [1m], request:
org.elasticsearch.action.bulk.BulkShardRequest@b25f027]]

Can you explain why is that?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · July 3, 2013, 5:17pm

This does not work. Use Cluster Update API after creating index for
changing index.number_of_shards and index.number_of_replicas

Example:

     ImmutableSettings.Builder settingsBuilder =

ImmutableSettings.settingsBuilder();
settingsBuilder.put("number_of_shards", 1);
settingsBuilder.put("number_of_replicas", 0);
UpdateSettingsRequest updateSettingsRequest = new
UpdateSettingsRequest("myindex")
.settings(settingsBuilder);
client.admin().indices().updateSettings(updateSettingsRequest).actionGet();

Where do you get your example code from?

Jörg

Am 03.07.13 00:21, schrieb vinod eligeti:

NodeBuilder nb = new NodeBuilder().clusterName("log-store");
nb.settings().put("node.data", true);
nb.settings().put("index.number_of_shards", 1);
nb.settings().put("index.number_of_replicas", 0);

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vinod_eligeti · July 3, 2013, 9:36pm

Thanks however, I donot want to update the settings for each index but
rather all indices. And another question I have is when I start the
embedded ES node I want to start with some settings and not like update the
settings as you have showed in the example.

BTW can you also explain why the bulk insertion is failing in the second
approach?

On Wed, Jul 3, 2013 at 10:17 AM, Jörg Prante joergprante@gmail.com wrote:

This does not work. Use Cluster Update API after creating index for
changing index.number_of_shards and index.number_of_replicas

Example:
    ImmutableSettings.Builder settingsBuilder = ImmutableSettings.**
settingsBuilder();
settingsBuilder.put("number_**of_shards", 1);
settingsBuilder.put("number_**of_replicas", 0);
UpdateSettingsRequest updateSettingsRequest = new
UpdateSettingsRequest("**myindex")
.settings(settingsBuilder);
client.admin().indices().**updateSettings(updateSettingsRequest).
actionGet();

Where do you get your example code from?

Jörg

Am 03.07.13 00:21, schrieb vinod eligeti:

NodeBuilder nb = new NodeBuilder().clusterName("**log-store");

nb.settings().put("node.data", true);
nb.settings().put("index.**number_of_shards", 1);
nb.settings().put("index.**number_of_replicas", 0);

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**2oqe8KhrGeM/unsubscribehttps://groups.google.com/d/topic/elasticsearch/2oqe8KhrGeM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.com elasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

jprante · July 4, 2013, 3:11pm

The settings for a node can be given like this:

Node node = NodeBuilder.nodeBuilder()
.settings(ImmutableSettings.settingsBuilder()
.put("index.number_of_shards",1)
.put("index.number_of_replicas", 0)
.build())
.build()
.clusterName("firstCluster")
.node();

More flexible is ImmutableSettings in the index creation request.

CreateIndexRequest indexRequest = new CreateIndexRequest("myindex");
indexRequest.settings(ImmutableSettings.settingsBuilder()
.put("index.number_of_shards",1)
.put("index.number_of_replicas", 0)
.build());
client.admin().indices().create(indexRequest).actionGet();

Another method is using config files.

Jörg

Am 03.07.13 23:36, schrieb vinod eligeti:

Thanks however, I donot want to update the settings for each index but
rather all indices. And another question I have is when I start the
embedded ES node I want to start with some settings and not like
update the settings as you have showed in the example.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

vinod_eligeti · July 8, 2013, 6:18pm

Thanks,

However I still see the same problem as I mentioned earlier in my first
post.

On Thu, Jul 4, 2013 at 8:11 AM, Jörg Prante joergprante@gmail.com wrote:

The settings for a node can be given like this:

Node node = NodeBuilder.nodeBuilder()
.settings(ImmutableSettings.**settingsBuilder()
.put("index.number_of_shards",**1)
.put("index.number_of_**replicas", 0)
.build())
.build()
.clusterName("firstCluster")
.node();

More flexible is ImmutableSettings in the index creation request.

CreateIndexRequest indexRequest = new CreateIndexRequest("myindex");
indexRequest.settings(**ImmutableSettings.**settingsBuilder()
.put("index.number_of_shards",**1)
.put("index.number_of_**replicas", 0)
.build());
client.admin().indices().**create(indexRequest).**actionGet();

Another method is using config files.

Jörg

Am 03.07.13 23:36, schrieb vinod eligeti:

Thanks however, I donot want to update the settings for each index but

rather all indices. And another question I have is when I start the
embedded ES node I want to start with some settings and not like update the
settings as you have showed in the example.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/**
topic/elasticsearch/**2oqe8KhrGeM/unsubscribehttps://groups.google.com/d/topic/elasticsearch/2oqe8KhrGeM/unsubscribe
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@**googlegroups.com elasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_out https://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.