Unassigned Shards Problem

Elastic Stack Elasticsearch
1

I have five nodes : Two Master Nodes, One Balancer Node, One Workhorse
Node, and One Coordinator Node.

I am shipping events from logstash, redis, to elasticsearch.

At the moment, my cluster is RED. The shards are created but no index is
created. I used to get an index like logstash.2014-05-22, but not anymore.

I deleted all my data, Cluster health goes GREEN.

However, as soon as data is sent from logstash -> redis -> elasticsearch,
my cluster health goes RED. I end up with unassigned shards. In my
/var/log/elasticsearch/logstash.log on my master, I see this in the log:

[2014-05-22 12:03:20,599][INFO ][cluster.metadata ] [Bora]
[logstash-2014.05.22] creating index, cause [auto(bulk api)], shards
[5]/[1], mappings [default]

On my master, this is the configuration:

cluster:
name: logstash
routing:
allocation:
awareness:
attributes: rack
node:
data: true
master: true

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "logstash",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 10
}

Is there an incorrect setting? I also installed ElasticHQ. It tells me the
same information.

Brian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/03c5974b-ae50-4f1c-9ba3-4ef94b564323%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

It does create an index, it says so in the log - [logstash-2014.05.22]
creating index - it's jut not assigning things.

You've set routing.allocation.awareness.attribute, but have you set the
node value, ie node.rack?
See

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 23 May 2014 02:22, Brian Wilkins bwilkins@gmail.com wrote:

I have five nodes : Two Master Nodes, One Balancer Node, One Workhorse
Node, and One Coordinator Node.

I am shipping events from logstash, redis, to elasticsearch.

At the moment, my cluster is RED. The shards are created but no index is
created. I used to get an index like logstash.2014-05-22, but not anymore.

I deleted all my data, Cluster health goes GREEN.

However, as soon as data is sent from logstash -> redis -> elasticsearch,
my cluster health goes RED. I end up with unassigned shards. In my
/var/log/elasticsearch/logstash.log on my master, I see this in the log:

[2014-05-22 12:03:20,599][INFO ][cluster.metadata ] [Bora]
[logstash-2014.05.22] creating index, cause [auto(bulk api)], shards
[5]/[1], mappings [default]

On my master, this is the configuration:

cluster:
name: logstash
routing:
allocation:
awareness:
attributes: rack
node:
data: true
master: true

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "logstash",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 10
}

Is there an incorrect setting? I also installed ElasticHQ. It tells me the
same information.

Brian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03c5974b-ae50-4f1c-9ba3-4ef94b564323%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/03c5974b-ae50-4f1c-9ba3-4ef94b564323%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEM624YGofrc55ND%3DXb3QW9cu799R0AKjB6_5Nmse_rMc9qcAw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for your reply. I set the node.rack to rack_one on all the nodes as
a test. In ElasticHQ, on the right it shows no indices. It is empty. In my
master, I see that the nodes are identifying with rack_one (all of them).

Any other clues?

Thanks

Brian

On Thursday, May 22, 2014 5:10:25 PM UTC-4, Mark Walkom wrote:

It does create an index, it says so in the log - [logstash-2014.05.22]
creating index - it's jut not assigning things.

You've set routing.allocation.awareness.attribute, but have you set the
node value, ie node.rack?
See
Elasticsearch Platform — Find real-time answers at scale | Elastic

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: ma...@campaignmonitor.com <javascript:>
web: www.campaignmonitor.com

On 23 May 2014 02:22, Brian Wilkins <bwil...@gmail.com <javascript:>>wrote:

I have five nodes : Two Master Nodes, One Balancer Node, One Workhorse
Node, and One Coordinator Node.

I am shipping events from logstash, redis, to elasticsearch.

At the moment, my cluster is RED. The shards are created but no index is
created. I used to get an index like logstash.2014-05-22, but not anymore.

I deleted all my data, Cluster health goes GREEN.

However, as soon as data is sent from logstash -> redis -> elasticsearch,
my cluster health goes RED. I end up with unassigned shards. In my
/var/log/elasticsearch/logstash.log on my master, I see this in the log:

[2014-05-22 12:03:20,599][INFO ][cluster.metadata ] [Bora]
[logstash-2014.05.22] creating index, cause [auto(bulk api)], shards
[5]/[1], mappings [default]

On my master, this is the configuration:

cluster:
name: logstash
routing:
allocation:
awareness:
attributes: rack
node:
data: true
master: true

curl -XGET 'http://localhost:9200/_cluster/health?pretty=true'
{
"cluster_name" : "logstash",
"status" : "red",
"timed_out" : false,
"number_of_nodes" : 5,
"number_of_data_nodes" : 3,
"active_primary_shards" : 0,
"active_shards" : 0,
"relocating_shards" : 0,
"initializing_shards" : 0,
"unassigned_shards" : 10
}

Is there an incorrect setting? I also installed ElasticHQ. It tells me
the same information.

Brian

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/03c5974b-ae50-4f1c-9ba3-4ef94b564323%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/03c5974b-ae50-4f1c-9ba3-4ef94b564323%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5bd4c3d5-3be5-44ef-a8a6-5dba6876130c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Went back and read the page again. So I made one master, workhorse, and balancer with rackid of rack_two for testing. One master shows rackid of rack_one. All nodes were restarted. The shards are still unassigned. Also,the indices in ElasticHQ are empty.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/4ac7b65e-3bc7-4f51-85e5-65ca3719880d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

I removed all the extra allocation stuff. When I did that, the shards were
all propogated. Health is green again.

On Thursday, May 22, 2014 6:56:24 PM UTC-4, Brian Wilkins wrote:

Went back and read the page again. So I made one master, workhorse, and
balancer with rackid of rack_two for testing. One master shows rackid of
rack_one. All nodes were restarted. The shards are still unassigned.
Also,the indices in ElasticHQ are empty.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b6e26a5c-7709-40e2-ae06-7c94e830ae2c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.