Elasticsearch deploy on ec2

Elastic Stack Elasticsearch
1

Hi,

I'm trying to deploy new elasticsearch environment on aws and the cluster
is always at red and i get the following message

{
"error": "ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state
not recovered / initialized];]",
"status": 503
}

I'm trying to deploy 3 master nodes, 3 data nodes, and 2 client nodes.
when i check cluster health:
{
"cluster_name": "stress_new",
"status": "red",
"timed_out": false,
"number_of_nodes": 8,
"number_of_data_nodes": 3,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}

please adivce

this is the configuration i used with changes by node role ( node.master &
node.data ):

cluster.name: stress_new
plugin.mandatory: cloud-aws
cloud.aws.access_key: ********
cloud.aws.secret_key: *********
cloud.aws.region: us-west-2
discovery.type: ec2
discovery.ec2.groups: stress_new_elasticsearch
discovery.ec2.host_type: private_ip
discovery.ec2.ping_timeout: 30s
discovery.ec2.tag.elasticsearch: stress_new

node.name: 172.******

node.master: false
node.data: false

index.number_of_shards: 1
index.number_of_replicas: 0
path.data: /mnt/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.mlockall: true

http.enabled: true

gateway.recover_after_nodes: 8
gateway.expected_nodes: 8
discovery.zen.minimum_master_nodes: 3
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false

index.search.slowlog.threshold.query.warn: 500ms
index.search.slowlog.threshold.query.info: 200ms
index.search.slowlog.threshold.query.debug: 100ms
index.search.slowlog.threshold.query.trace: 50ms

index.search.slowlog.threshold.fetch.warn: 500ms
index.search.slowlog.threshold.fetch.info: 200ms
index.search.slowlog.threshold.fetch.debug: 100ms
index.search.slowlog.threshold.fetch.trace: 50ms

index.indexing.slowlog.threshold.index.warn: 500ms
index.indexing.slowlog.threshold.index.info: 200ms
index.indexing.slowlog.threshold.index.debug: 1000ms
index.indexing.slowlog.threshold.index.trace: 50ms

script.disable_dynamic: false
script.native.socialScoreCalc.type: *****.SocialScriptFactory
script.default_lang: native

action.disable_delete_all_indices: true
action.auto_create_index: .marvel-*
indices.fielddata.cache.size: 30%
indices.fielddata.cache.expire: 15s
marvel.agent.enabled: false
allow_leading_wildcard: false
script.groovy.sandbox.enabled: false

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/843e39bd-842a-4301-899d-7a1bb1d119a9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Can you describe your deployment process? the cluster can't be always red

  • it should be green when you first deploy

Other than that, check the obvious - that AWS securty groups are properly
defined for those machines (all of them under the same named security group)

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:11 PM, Eliran Shlomo eliran@whipclip.com wrote:

Hi,

I'm trying to deploy new elasticsearch environment on aws and the cluster
is always at red and i get the following message

{
"error": "ClusterBlockException[blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];]",
"status": 503
}

I'm trying to deploy 3 master nodes, 3 data nodes, and 2 client nodes.
when i check cluster health:
{
"cluster_name": "stress_new",
"status": "red",
"timed_out": false,
"number_of_nodes": 8,
"number_of_data_nodes": 3,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}

please adivce

this is the configuration i used with changes by node role ( node.master &
node.data ):

cluster.name: stress_new
plugin.mandatory: cloud-aws
cloud.aws.access_key: ********
cloud.aws.secret_key: *********
cloud.aws.region: us-west-2
discovery.type: ec2
discovery.ec2.groups: stress_new_elasticsearch
discovery.ec2.host_type: private_ip
discovery.ec2.ping_timeout: 30s
discovery.ec2.tag.elasticsearch: stress_new

node.name: 172.******

node.master: false
node.data: false

index.number_of_shards: 1
index.number_of_replicas: 0
path.data: /mnt/elasticsearch

path.logs: /var/log/elasticsearch

bootstrap.mlockall: true

http.enabled: true

gateway.recover_after_nodes: 8
gateway.expected_nodes: 8
discovery.zen.minimum_master_nodes: 3
discovery.zen.ping.timeout: 10s
discovery.zen.ping.multicast.enabled: false

index.search.slowlog.threshold.query.warn: 500ms
index.search.slowlog.threshold.query.info: 200ms
index.search.slowlog.threshold.query.debug: 100ms
index.search.slowlog.threshold.query.trace: 50ms

index.search.slowlog.threshold.fetch.warn: 500ms
index.search.slowlog.threshold.fetch.info: 200ms
index.search.slowlog.threshold.fetch.debug: 100ms
index.search.slowlog.threshold.fetch.trace: 50ms

index.indexing.slowlog.threshold.index.warn: 500ms
index.indexing.slowlog.threshold.index.info: 200ms
index.indexing.slowlog.threshold.index.debug: 1000ms
index.indexing.slowlog.threshold.index.trace: 50ms

script.disable_dynamic: false
script.native.socialScoreCalc.type: *****.SocialScriptFactory
script.default_lang: native

action.disable_delete_all_indices: true
action.auto_create_index: .marvel-*
indices.fielddata.cache.size: 30%
indices.fielddata.cache.expire: 15s
marvel.agent.enabled: false
allow_leading_wildcard: false
script.groovy.sandbox.enabled: false

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/843e39bd-842a-4301-899d-7a1bb1d119a9%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/843e39bd-842a-4301-899d-7a1bb1d119a9%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZstAUwTELHxOx%3DqWBuOjByEn3mhgdd93-kG3M%3DxQMSS7g%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi,
Since the first moment the cluster in in red
The servers are under the same security group and inside the security group i allow any/any between the servers.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ed1d0f69-93cc-4467-a0ad-bffe4b5175df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Wait a second, you should use gateway.expected_data_nodes: 3 and
gateway.expected_master_nodes: 3 instead of what you have there now. Also
min master nodes should be 2 in your case.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:22 PM, Eliran Shlomo eliran@whipclip.com wrote:

Hi,
Since the first moment the cluster in in red
The servers are under the same security group and inside the security
group i allow any/any between the servers.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/ed1d0f69-93cc-4467-a0ad-bffe4b5175df%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtJEVnyXCDkEX7Hj%3DR%3Dvyy%3DUsUiiDSXC0EVhBMU2PaYXQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

"Wait a second, you should use gateway.expected_data_nodes: 3 and gateway.expected_master_nodes: 3 instead of what you have there now. Also min master nodes should be 2 in your case."

Those settings should be in the configuration of all nodes or only in the external gateway?(client)

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/e6914e16-28ac-4d00-96b7-b3b007538aba%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Master eligible nodes and Data nodes need to have this setting

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:31 PM, Eliran Shlomo eliran@whipclip.com wrote:

"Wait a second, you should use gateway.expected_data_nodes: 3 and
gateway.expected_master_nodes: 3 instead of what you have there now. Also
min master nodes should be 2 in your case."

Those settings should be in the configuration of all nodes or only in the
external gateway?(client)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e6914e16-28ac-4d00-96b7-b3b007538aba%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtzjXeHrzHiRAitfOHASpXrvzXza8JPVObtjMwCHqMrVA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

7

Hi, made the changes.
no change in the cluster status and the same response from the servers

{
"cluster_name": "stress_new",
"status": "red",
"timed_out": false,
"number_of_nodes": 8,
"number_of_data_nodes": 3,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}

get _status

{
"error": "ClusterBlockException[blocked by: [SERVICE_UNAVAILABLE/1/state
not recovered / initialized];]",
"status": 503
}

On Monday, February 16, 2015 at 8:53:11 PM UTC+2, Itamar Syn-Hershko wrote:

Master eligible nodes and Data nodes need to have this setting

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:31 PM, Eliran Shlomo <eli...@whipclip.com
<javascript:>> wrote:

"Wait a second, you should use gateway.expected_data_nodes: 3 and
gateway.expected_master_nodes: 3 instead of what you have there now. Also
min master nodes should be 2 in your case."

Those settings should be in the configuration of all nodes or only in the
external gateway?(client)

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/e6914e16-28ac-4d00-96b7-b3b007538aba%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cd40be93-9b10-4bf1-ac9a-eafe8b98c8fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

8

Remove the number of nodes setting, if that doesn't help start looking at
the logs. I seen clusters on AWS that took some time to discover and
stabilize, it may also be that.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:56 PM, Eliran Shlomo eliran@whipclip.com wrote:

Hi, made the changes.
no change in the cluster status and the same response from the servers

{
"cluster_name": "stress_new",
"status": "red",
"timed_out": false,
"number_of_nodes": 8,
"number_of_data_nodes": 3,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}

get _status

{
"error": "ClusterBlockException[blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];]",
"status": 503
}

On Monday, February 16, 2015 at 8:53:11 PM UTC+2, Itamar Syn-Hershko wrote:

Master eligible nodes and Data nodes need to have this setting

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:31 PM, Eliran Shlomo eli...@whipclip.com
wrote:

"Wait a second, you should use gateway.expected_data_nodes: 3 and
gateway.expected_master_nodes: 3 instead of what you have there now. Also
min master nodes should be 2 in your case."

Those settings should be in the configuration of all nodes or only in
the external gateway?(client)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/e6914e16-28ac-4d00-96b7-b3b007538aba%
40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cd40be93-9b10-4bf1-ac9a-eafe8b98c8fe%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cd40be93-9b10-4bf1-ac9a-eafe8b98c8fe%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4Zsa0DgkCJuuLrhyrssM%2BEFMQ4jcZwECWmq%3Dq0GFyqHVEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

9

Tnx for the help Itamar.
solved by changing gateway.recover_after_nodes value from 8 to 6
i guess client node shouldn't be count for recover

On Monday, February 16, 2015 at 8:59:48 PM UTC+2, Itamar Syn-Hershko wrote:

Remove the number of nodes setting, if that doesn't help start looking at
the logs. I seen clusters on AWS that took some time to discover and
stabilize, it may also be that.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:56 PM, Eliran Shlomo <eli...@whipclip.com
<javascript:>> wrote:

Hi, made the changes.
no change in the cluster status and the same response from the servers

{
"cluster_name": "stress_new",
"status": "red",
"timed_out": false,
"number_of_nodes": 8,
"number_of_data_nodes": 3,
"active_primary_shards": 0,
"active_shards": 0,
"relocating_shards": 0,
"initializing_shards": 0,
"unassigned_shards": 0
}

get _status

{
"error": "ClusterBlockException[blocked by:
[SERVICE_UNAVAILABLE/1/state not recovered / initialized];]",
"status": 503
}

On Monday, February 16, 2015 at 8:53:11 PM UTC+2, Itamar Syn-Hershko
wrote:

Master eligible nodes and Data nodes need to have this setting

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Lucene.NET committer and PMC member

On Mon, Feb 16, 2015 at 8:31 PM, Eliran Shlomo eli...@whipclip.com
wrote:

"Wait a second, you should use gateway.expected_data_nodes: 3 and
gateway.expected_master_nodes: 3 instead of what you have there now. Also
min master nodes should be 2 in your case."

Those settings should be in the configuration of all nodes or only in
the external gateway?(client)

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/e6914e16-28ac-4d00-96b7-b3b007538aba%
40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/cd40be93-9b10-4bf1-ac9a-eafe8b98c8fe%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/cd40be93-9b10-4bf1-ac9a-eafe8b98c8fe%40googlegroups.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9473e8bf-def6-4f04-89f3-b957394b4cf4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.