Unable to make Cluster state to Green

anusha6 · January 20, 2016, 7:45am

Hello Team,
I am using two nodes in same host but am unable to see the cluster state as green,

Here is my response for cluster health
GET /_cluster/health?pretty=true

{
"cluster_name": "elasticsearch",
"status": "yellow",
"timed_out": false,
"number_of_nodes": 2,
"number_of_data_nodes": 2,
"active_primary_shards": 51,
"active_shards": 51,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 51,
"number_of_pending_tasks": 0
}

Can anyone suggest me on this?????

dadoonet · January 20, 2016, 7:58am

You probably don't have enough disk space.
I think you can see it in logs...

anusha6 · January 20, 2016, 8:07am

Hello David,

Here are my logs..

[2016-01-19 17:09:36,614][INFO ][node                     ] [Franz Kafka1] initializing ...
[2016-01-19 17:09:36,784][INFO ][com.floragunn.searchguard.SearchGuardPlugin] Class enhancements for DLS/FLS successful
[2016-01-19 17:09:36,786][INFO ][plugins                  ] [Franz Kafka1] loaded [analysis-phonetic, searchguard (disabled)], sites [migration]
[2016-01-19 17:09:39,954][INFO ][node                     ] [Franz Kafka1] initialized
[2016-01-19 17:09:40,011][INFO ][node                     ] [Franz Kafka1] starting ...
[2016-01-19 17:09:40,071][INFO ][transport                ] [Franz Kafka1] bound_address {inet[/192.168.1.59:9300]}, publish_address {inet[/192.168.1.59:9300]}
[2016-01-19 17:09:40,079][INFO ][discovery                ] [Franz Kafka1] elasticsearch/SPI6MqvpREmv6ewlmO1rGQ
[2016-01-19 17:09:44,100][INFO ][cluster.service          ] [Franz Kafka1] new_master [Franz Kafka1][SPI6MqvpREmv6ewlmO1rGQ][Anusha-PC][inet[/192.168.1.59:9300]]{master=true}, reason: zen-disco-join (elected_as_master)
[2016-01-19 17:09:44,164][INFO ][http                     ] [Franz Kafka1] bound_address {inet[/192.168.1.59:9200]}, publish_address {inet[/192.168.1.59:9200]}
[2016-01-19 17:09:44,165][INFO ][node                     ] [Franz Kafka1] started
[2016-01-19 17:09:44,706][INFO ][cluster.routing.allocation.decider] [Franz Kafka1] updating [cluster.routing.allocation.disable_allocation] from [false] to [true]
[2016-01-19 17:09:44,707][INFO ][indices.ttl              ] [Franz Kafka1] updating indices.ttl.interval from [1m] to [15s]
[2016-01-19 17:09:44,814][INFO ][gateway                  ] [Franz Kafka1] recovered [11] indices into cluster_state
[2016-01-19 17:10:00,179][INFO ][cluster.service          ] [Franz Kafka1] added {[Franz Kafka2][UqFUB2KbQJa8r6ygz2s4zw][Gowthami-PC][inet[/192.168.1.45:9300]]{master=true},}, reason: zen-disco-receive(join from node[[Franz Kafka2][UqFUB2KbQJa8r6ygz2s4zw][Gowthami-PC][inet[/192.168.1.45:9300]]{master=true}])
[2016-01-19 17:12:27,863][INFO ][cluster.service          ] [Franz Kafka1] removed {[Franz Kafka2][UqFUB2KbQJa8r6ygz2s4zw][Gowthami-PC][inet[/192.168.1.45:9300]]{master=true},}, reason: zen-disco-node_left([Franz Kafka2][UqFUB2KbQJa8r6ygz2s4zw][Gowthami-PC][inet[/192.168.1.45:9300]]{master=true})
[2016-01-19 19:00:33,891][INFO ][bootstrap                ] running graceful exit on windows
[2016-01-19 19:00:33,893][INFO ][node                     ] [Franz Kafka1] stopping ...
[2016-01-19 19:00:33,997][INFO ][node                     ] [Franz Kafka1] stopped
[2016-01-19 19:00:33,997][INFO ][node                     ] [Franz Kafka1] closing ...
[2016-01-19 19:00:34,000][INFO ][node                     ] [Franz Kafka1] closed

Am unable to find disk space in logs....

anusha6 · January 20, 2016, 8:18am

Hello David, here are my indexes and the space used by them, I hope this data I useful to you,

health status index                                pri rep     docs.count    docs.deleted     store.size   pri.store.size 
yellow open   othermotive_parttype     5    1        435447            0                       73.9mb         73.9mb 
yellow open   dcl_raac_json                    5   1             9366             0                          1.8mb          1.8mb 
yellow open   master_index                   5   1                    0             0                             575b           575b 
yellow open   pdlfull                                5   1             5072             0                               1mb            1mb 
yellow open   dcl_raac_pcdb                  5   1            20241            0                             9.7mb          9.7mb 
yellow open   ymme                                5   1            35451            0                            25.3mb         25.3mb 
yellow open   demoymme                      5   1            35451            0                            16.8mb         16.8mb 
yellow open   ast                                      5   1                 336            0                            65.6kb         65.6kb 
yellow open   phonetic_sample             5   1                 314            0                            70.5kb         70.5kb 
yellow open   twitter                                1   1                 314            0                            31.7kb         31.7kb 
yellow open   othermotive_ymme        5   1             14396            0                              9.6mb          9.6mb

dadoonet · January 20, 2016, 8:38am

It shows that your second node has been removed 2 minutes after it appeared.

Why did you stop it?
Why did you stop the first node as well?

When did you run the cluster health? In the mean time?

anusha6 · January 20, 2016, 9:35am

No that was like my last trail to make a two node cluster, normally I just pasted the logs that am getting when two nodes are connected..
that's it , am getting only those logs apart from the start and stop logs..

I have started two nodes and verified the cluster state got as yellow.....
Do I need to do anything more than this???

anusha6 · January 20, 2016, 9:41am

For you to know that how much space occupied by the indexes I pasted the indexes and the sizes they occupied,
So is that becaz of the more disk space occupied or due to any other reason???

And Yes I run the cluster health in the mean while..

dadoonet · January 20, 2016, 10:04am

Well the only time I saw my replicas not being allocated was when I was below 20% of available disk space.

I would:

start node 1 again
wait for yellow status
start node 2
call https://www.elastic.co/guide/en/elasticsearch/reference/current/cat-pending-tasks.html

anusha6 · January 20, 2016, 10:22am

I have my cluster settings as shown,
{ "persistent": {
"cluster": {
"routing": {
"allocation": {
"disable_allocation": "true"
}
}
},
"discovery": {
"zen": {
"publish_timeout": "30s"
}
},
"indices": {
"ttl": {
"interval": "15000"
}
}
},
"transient": {}
}

is it the cause????

anusha6 · January 20, 2016, 10:39am

Hello David,
I stopped two nodes and started them again, this time am getting few indexes as yellow and few are green

And am getting exception as

[2016-01-20 16:03:29,676][WARN ][indices.cluster ] [Franz Kafka2] [[phonetic_sample][0]] marking and sending shard failed due to [failed to create index]
org.elasticsearch.indices.IndexCreationException: [phonetic_sample] failed to create index
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:330)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewIndices(IndicesClusterStateService.java:311)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:180)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:467)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: failed to find token filter type [phonetic] for [my_metaphone]
at org.elasticsearch.index.analysis.AnalysisModule.configure(AnalysisModule.java:249)
at org.elasticsearch.common.inject.AbstractModule.configure(AbstractModule.java:60)
at org.elasticsearch.common.inject.spi.Elements$RecordingBinder.install(Elements.java:204)

... 8 more

Caused by: org.elasticsearch.common.settings.NoClassSettingsException: Failed to load class setting [type] with value [phonetic]
at org.elasticsearch.common.settings.ImmutableSettings.loadClass(ImmutableSettings.java:476)
at org.elasticsearch.common.settings.ImmutableSettings.getAsClass(ImmutableSettings.java:464)
at org.elasticsearch.index.analysis.AnalysisModule.configure(AnalysisModule.java:239)
... 16 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.index.analysis.phonetic.PhoneticTokenFilterFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at org.elasticsearch.common.settings.ImmutableSettings.loadClass(ImmutableSettings.java:474)
... 18 more
[2016-01-20 16:03:29,686][WARN ][indices.cluster ] [Franz Kafka2] [[phonetic_sample][3]] marking and sending shard failed due to [failed to create index]
org.elasticsearch.indices.IndexCreationException: [phonetic_sample] failed to create index
at org.elasticsearch.indices.IndicesService.createIndex(IndicesService.java:330)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.applyNewIndices(IndicesClusterStateService.java:311)
at org.elasticsearch.indices.cluster.IndicesClusterStateService.clusterChanged(IndicesClusterStateService.java:180)
at org.elasticsearch.cluster.service.InternalClusterService$UpdateTask.run(InternalClusterService.java:467)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.runAndClean(PrioritizedEsThreadPoolExecutor.java:188)
at org.elasticsearch.common.util.concurrent.PrioritizedEsThreadPoolExecutor$TieBreakingPrioritizedRunnable.run(PrioritizedEsThreadPoolExecutor.java:158)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
Caused by: org.elasticsearch.ElasticsearchIllegalArgumentException: failed to find token filter type [phonetic] for [my_metaphone]
at org.elasticsearch.index.analysis.AnalysisModule.configure(AnalysisModule.java:249)
... 8 more
Caused by: org.elasticsearch.common.settings.NoClassSettingsException: Failed to load class setting [type] with value [phonetic]
at org.elasticsearch.index.analysis.AnalysisModule.configure(AnalysisModule.java:239)
... 16 more
Caused by: java.lang.ClassNotFoundException: org.elasticsearch.index.analysis.phonetic.PhoneticTokenFilterFactory
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
... 18 more

Why suddenly the behaviour changed from unassigned to assign of few shards and few got exceptions?????

dadoonet · January 20, 2016, 10:39am

Probably you did not install the phonetic plugin on the second node?

anusha6 · January 20, 2016, 10:54am

Thanks David, Installed plugin works fine all went to Green...

Topic		Replies	Views
How to fix cluster health yellow with Elastic Search Elasticsearch	5	51141	July 6, 2017
Status is green but should be yellow Elasticsearch	5	394	July 6, 2017
How did we get into a yellow state and how do we get out of there? Elasticsearch	7	321	July 6, 2017
Unable to make my cluster state as Green even with more than 1 nodes Elasticsearch	5	695	July 5, 2017
Cluster stuck in a yellow state Elasticsearch	16	8663	July 5, 2017

Unable to make Cluster state to Green

Related topics