Elasticsearch not assigning replica shards and not reallocating

I have an issue with elasticsearch version 1.4.1, i had indexes without
replicas and i recently i added 1 replica to all indexes and they are not
getting allocated.
I also have an issue when using cluster setting to exclude a node from
allocation, it just doesnt work, data is not moved from the node, this used
to work on 1.3.2.

I don't know what could be wrong and i dont see anything meaningful in logs.

Can someone help me?

Thanks,
Sebastian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/51d4d2f5-026d-43e2-9087-927ba7279067%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

How many nodes do you have?

Can you provide the command you are sending ES to set the exclude?

On 4 December 2014 at 13:46, Sebastián Schepens <
sebastian.schepens@mercadolibre.com> wrote:

I have an issue with elasticsearch version 1.4.1, i had indexes without
replicas and i recently i added 1 replica to all indexes and they are not
getting allocated.
I also have an issue when using cluster setting to exclude a node from
allocation, it just doesnt work, data is not moved from the node, this used
to work on 1.3.2.

I don't know what could be wrong and i dont see anything meaningful in
logs.

Can someone help me?

Thanks,
Sebastian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/51d4d2f5-026d-43e2-9087-927ba7279067%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/51d4d2f5-026d-43e2-9087-927ba7279067%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8e_%3DVBmh21pknod7ruMM0e5Y0hhsRJ3igtb6_SUNxODw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

I have 25 nodes and 12 java clients that bulk index.
This is the command i'm using:

curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.exclude._host" : "HOST"
}
}'

Got any idea why replica shards are not being allocated?

This is my config file:
indices.fielddata.cache.size: 30%
indices.fielddata.cache.expire: 10m
indices.breaker.fielddata.limit: 60%

cluster.name: elasticsearch
action.auto_create_index: true

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: .85
cluster.routing.allocation.disk.watermark.high: .90

node.name: "##HOSTNAME##"
node.master: false
node.data: true
node.client: false

index.number_of_shards: 5
index.number_of_replicas: 1

bootstrap.mlockall: true

transport.tcp.compress: true
http.port: 8080

gateway.type: local
gateway.recover_after_nodes: 2
gateway.recover_after_time: 1m
gateway.expected_nodes: ##SERVERS_COUNT##

cluster.routing.allocation.node_initial_primaries_recoveries: 20
cluster.routing.allocation.node_concurrent_recoveries: 10
indices.recovery.max_bytes_per_sec: 50mb
indices.recovery.concurrent_streams: 5

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ##MASTER_NODES##

processors: 16
index.mapper.dynamic: true
action.disable_delete_all_indices: true
index.refresh_internal: 10s
cluster.routing.allocation.cluster_concurrent_rebalance: 20
script.disable_dynamic: true
http.compression: true

Thanks,
Sebastian

On Thursday, December 4, 2014 1:00:24 AM UTC-3, Mark Walkom wrote:

How many nodes do you have?

Can you provide the command you are sending ES to set the exclude?

On 4 December 2014 at 13:46, Sebastián Schepens <
sebastian...@mercadolibre.com <javascript:>> wrote:

I have an issue with elasticsearch version 1.4.1, i had indexes without
replicas and i recently i added 1 replica to all indexes and they are not
getting allocated.
I also have an issue when using cluster setting to exclude a node from
allocation, it just doesnt work, data is not moved from the node, this used
to work on 1.3.2.

I don't know what could be wrong and i dont see anything meaningful in
logs.

Can someone help me?

Thanks,
Sebastian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/51d4d2f5-026d-43e2-9087-927ba7279067%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/51d4d2f5-026d-43e2-9087-927ba7279067%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/18fb961b-a443-489c-9d97-015e351728ac%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Nevermind, i think i fixed it, it seems somewhere between 1.3.2 and 1.4.0
config settingts for watermark changed, i changed this:
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%

for:
cluster.routing.allocation.disk.watermark.low: 85%
cluster.routing.allocation.disk.watermark.high: 90%

And it worked!

On Thursday, December 4, 2014 10:27:07 AM UTC-3, Sebastián Schepens wrote:

I have 25 nodes and 12 java clients that bulk index.
This is the command i'm using:

curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.exclude._host" : "HOST"
}
}'

Got any idea why replica shards are not being allocated?

This is my config file:
indices.fielddata.cache.size: 30%
indices.fielddata.cache.expire: 10m
indices.breaker.fielddata.limit: 60%

cluster.name: elasticsearch
action.auto_create_index: true

cluster.routing.allocation.disk.threshold_enabled: true
cluster.routing.allocation.disk.watermark.low: .85
cluster.routing.allocation.disk.watermark.high: .90

node.name: "##HOSTNAME##"
node.master: false
node.data: true
node.client: false

index.number_of_shards: 5
index.number_of_replicas: 1

bootstrap.mlockall: true

transport.tcp.compress: true
http.port: 8080

gateway.type: local
gateway.recover_after_nodes: 2
gateway.recover_after_time: 1m
gateway.expected_nodes: ##SERVERS_COUNT##

cluster.routing.allocation.node_initial_primaries_recoveries: 20
cluster.routing.allocation.node_concurrent_recoveries: 10
indices.recovery.max_bytes_per_sec: 50mb
indices.recovery.concurrent_streams: 5

discovery.zen.minimum_master_nodes: 2
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ##MASTER_NODES##

processors: 16
index.mapper.dynamic: true
action.disable_delete_all_indices: true
index.refresh_internal: 10s
cluster.routing.allocation.cluster_concurrent_rebalance: 20
script.disable_dynamic: true
http.compression: true

Thanks,
Sebastian

On Thursday, December 4, 2014 1:00:24 AM UTC-3, Mark Walkom wrote:

How many nodes do you have?

Can you provide the command you are sending ES to set the exclude?

On 4 December 2014 at 13:46, Sebastián Schepens <
sebastian...@mercadolibre.com> wrote:

I have an issue with elasticsearch version 1.4.1, i had indexes without
replicas and i recently i added 1 replica to all indexes and they are not
getting allocated.
I also have an issue when using cluster setting to exclude a node from
allocation, it just doesnt work, data is not moved from the node, this used
to work on 1.3.2.

I don't know what could be wrong and i dont see anything meaningful in
logs.

Can someone help me?

Thanks,
Sebastian

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/51d4d2f5-026d-43e2-9087-927ba7279067%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/51d4d2f5-026d-43e2-9087-927ba7279067%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b40d0fb2-e1d7-4686-be2d-c6ad5c6ea62d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.