# Unable to restart elasticsearch

**URL:** <https://discuss.elastic.co/t/unable-to-restart-elasticsearch/270665>\
**Category:** Elasticsearch\
**Created:** [April 20, 2021, 7:25am UTC](https://discuss.elastic.co/t/unable-to-restart-elasticsearch/270665 "2021-04-20T07:25:17Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![sarabande](https://avatars.discourse-cdn.com/v4/letter/s/7cd45c/32.png) [@sarabande](https://discuss.elastic.co/u/sarabande)\
**Post date:** [April 20, 2021, 7:25am UTC](https://discuss.elastic.co/t/unable-to-restart-elasticsearch/270665/1 "2021-04-20T07:25:17Z")

</div>

Hi,

I'm using elasticsearch in a multi-node cluster.

Here's the configuration file (elasticsearch.yml) from one of the two elasticsearch log master nodes (in the cluster I have also 6 elasticsearch log nodes):

```auto
# Network
network:
  host: 0.0.0.0
  publish_host: c4t19154.xxx.xxx
http.port: 9200

# Cluster / node name
cluster.name: "es-sa20-xxx-log-hdp-itg-h4"
node.name: "c4t19154.xxx.xxx"

# Node role
node.master: True
node.data: False

# Discovery
discovery.seed_hosts: ['c4t19156.xxx.xxx:9300', 'c4t19154.xxx.xxx:9300', 'c4t19145.xxx.xxx:9300', 'c4t19147.xxx.xxx:9300', 'c4t19867.xxx.xxx:9300', 'c4t19149.xxx.xxx:9300', 'c4t19150.xxx.xxx:9300', 'hc9t09679.xxx.xxx:9300']
cluster.initial_master_nodes: ['c4t19156.xxx.xxx', 'c4t19154.xxx.xxx']

# Memory - Performance
bootstrap.memory_lock: true

# Monitoring
xpack.monitoring.collection.enabled: true
xpack.monitoring.exporters:
  csa_monitoring:
    type: http
    host: ['https://c9t24539.xxx.xxx:9200', 'https://c9t24540.xxx.xxx:9200', 'https://c9t24541.xxx.xxx:9200']
    auth:
      username: "elastic"
      password: "changeme"
    ssl:
      certificate_authorities: ["/usr/share/elasticsearch/config/XXX_ENT_Private_SSL_CA_bundle.crt"]

# Security
xpack.security.enabled: true
xpack.security.audit.enabled: false

# TLS/SSL
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.key: "/usr/share/elasticsearch/config/sa-monitoring-itg-h4.xxx.xxx.key"
xpack.security.transport.ssl.certificate: "/usr/share/elasticsearch/config/sa-monitoring-itg-h4.xxx.xxx.crt"
xpack.security.transport.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/XXX_ENT_Private_SSL_CA_bundle.crt"]

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key: "/usr/share/elasticsearch/config/sa-monitoring-itg-h4.xxx.xxx.key"
xpack.security.http.ssl.certificate: "/usr/share/elasticsearch/config/sa-monitoring-itg-h4.xxx.xxx.crt"
xpack.security.http.ssl.certificate_authorities: ["/usr/share/elasticsearch/config/XXX_ENT_Private_SSL_CA_bundle.crt"]

# Authentication - Authorization
xpack.security.authc:
  anonymous:
    username: anonymous_user
    roles: superuser
    authz_exception: true
  realms:
    native.native1:
      order: 0

# Watcher mail config
xpack.notification.email.account:
  exchange_account:
    profile: outlook
    email_defaults:
      from: 'elasticsearch-logs on hdp-itg-h4 <systemteams-tss-rnd@xxx.flowdock.com>'
    smtp:
      starttls.enable: true
      host: smtp1.xxx.com
      port: 25

#Enable OIDC token service
xpack.security.authc.token.enabled: true

##Create an OpenID Connect realm
xpack.security.authc.realms.oidc.uidoidc:
  order: 1
  rp.client_id: "sa-kibana-itg"
  rp.response_type: code
  rp.redirect_uri: "https://sa-monitoring-itg-h4.xxx.xxx/kibana-api/security/v1/oidc"
  op.issuer: "https://login-itg.ext.xxx.com"
  op.authorization_endpoint: "https://login-itg.ext.xxx.com/as/authorization.oauth2"
  op.token_endpoint: "https://login-itg.ext.xxx.com/as/token.oauth2"
  op.jwkset_path: "https://login-itg.ext.xxx.com/pf/JWKS"
  op.userinfo_endpoint: "https://login-itg.ext.xxx.com/idp/userinfo.openid"
  op.endsession_endpoint: "https://login-itg.ext.xxx.com/idp/startSLO.ping"
  rp.post_logout_redirect_uri: "https://sa-monitoring-itg-h4.xxx.xxx/logged_out"
  rp.signature_algorithm: HS256
  claims.principal: uid
  #claims.groups: "http://example.info/claims/groups"

```

I'm using the following ansible play to restart elasticsearch:

```auto
---

- name: Stop elasticsearch docker container
  docker_container:
    name: elasticsearch
    image: "{{ it_dtr_host }}/sa20sre/elasticsearch"
    state: stopped
  ignore_errors: yes

- name: Add elasticsearch data dir
  file:
    path: "{{base_path}}/elasticsearch/data"
    state: directory
    mode: 0777

- name: Set Facts
  include: set_facts.yml

- name: Start elasticsearch docker container
  docker_container:
    name: elasticsearch
    image: "{{ it_dtr_host }}/sa20sre/elasticsearch:{{ elasticsearch_version }}"
    state: started
  ignore_errors: yes

```

After the last task (Start elasticsearch docker container), elasticsearch docker containers are started on all nodes but after aproximatelly 30 seconds all of them stop with the following message in the logs:

```auto
{"type": "server", "timestamp": "2021-04-08T13:39:58,777Z", "level": "INFO", "component": "o.e.t.TransportService", "cluster.name": "docker-cluster", "node.name": "0d7781abc420", "message": "publish_address {172.17.0.4:9300}, bound_addresses {0.0.0.0:9300}" }
{"type": "server", "timestamp": "2021-04-08T13:39:58,790Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "0d7781abc420", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
ERROR: [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
{"type": "server", "timestamp": "2021-04-08T13:39:58,887Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0d7781abc420", "message": "stopping ..." }
{"type": "server", "timestamp": "2021-04-08T13:39:59,076Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0d7781abc420", "message": "stopped" }
{"type": "server", "timestamp": "2021-04-08T13:39:59,077Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0d7781abc420", "message": "closing ..." }
{"type": "server", "timestamp": "2021-04-08T13:39:59,217Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "0d7781abc420", "message": "closed" }

```

The log file says that the default discovery settings are unsuitable for production use and that at least one of discovery.seed\_hosts, discovery.seed\_providers and cluster.initial\_master\_nodes must be configured. As you can see from the configuration I have two of them configured (discovery.seed\_hosts and cluster.initial\_master\_nodes).

Starting the elasticsearch manually doesn't help, elasticsearch starts and stops after a short period of time with the same error.

I've turned the net upside down trying to find something usefull but nothing I tried helped.

I've even tried to resolve this by setting es.enforce.bootstrap.checks to false in jvm.options file (although this is not a single node cluster) but it didn't help:

```auto
# avoid bootstrap checks
-Des.enforce.bootstrap.checks=false

```

I'll appreciate any help with this.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 20, 2021, 8:43am UTC](https://discuss.elastic.co/t/unable-to-restart-elasticsearch/270665/2 "2021-04-20T08:43:12Z")

</div>

It seems the nodes are not reading the `elasticsearch.yml` file you think they are. Not only does it say that `discovery.seed_hosts` is not set, but also:

> [@sarabande](#):
>
> `"cluster.name": "docker-cluster", "node.name": "0d7781abc420"`

This isn't the cluster name or the node name you have configured.

---

<div class="post-metadata">

**Author:** ![sarabande](https://avatars.discourse-cdn.com/v4/letter/s/7cd45c/32.png) [@sarabande](https://discuss.elastic.co/u/sarabande)\
**Post date:** [April 20, 2021, 9:56am UTC](https://discuss.elastic.co/t/unable-to-restart-elasticsearch/270665/3 "2021-04-20T09:56:38Z")

</div>

@DavidTurner Thank you for promptly answering my question.

I have changed the play for starting the elasticsearch docker container. It seems that now the node.name is reported correctly:

```auto
{"type": "server", "timestamp": "2021-04-20T09:50:15,940Z", "level": "INFO", "component": "o.e.b.BootstrapChecks", "cluster.name": "docker-cluster", "node.name": "c4t19156.xxx.xxx", "message": "bound or publishing to a non-loopback address, enforcing bootstrap checks" }
ERROR: [1] bootstrap checks failed
[1]: the default discovery settings are unsuitable for production use; at least one of [discovery.seed_hosts, discovery.seed_providers, cluster.initial_master_nodes] must be configured
{"type": "server", "timestamp": "2021-04-20T09:50:15,973Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c4t19156.xxx.xxx", "message": "stopping ..." }
{"type": "server", "timestamp": "2021-04-20T09:50:16,028Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c4t19156.xxx.xxx", "message": "stopped" }
{"type": "server", "timestamp": "2021-04-20T09:50:16,028Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c4t19156.xxx.xxx", "message": "closing ..." }
{"type": "server", "timestamp": "2021-04-20T09:50:16,063Z", "level": "INFO", "component": "o.e.n.Node", "cluster.name": "docker-cluster", "node.name": "c4t19156.xxx.xxx", "message": "closed" }
{"type": "server", "timestamp": "2021-04-20T09:50:16,070Z", "level": "INFO", "component": "o.e.x.m.p.NativeController", "cluster.name": "docker-cluster", "node.name": "c4t19156.xxx.xxx", "message": "Native controller process has stopped - no new native processes can be started" }

```

But the problem still exists. Checking if I need also to set docker-cluster.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [April 20, 2021, 10:09am UTC](https://discuss.elastic.co/t/unable-to-restart-elasticsearch/270665/4 "2021-04-20T10:09:56Z")

</div>

TBC the problem wasn't that the node name was wrong, it was that Elasticsearch isn't using the `elasticsearch.yml` file you shared. It still isn't, because the `cluster.name` is wrong and it says that `discovery.seed_hosts` is not set.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 18, 2021, 10:10am UTC](https://discuss.elastic.co/t/unable-to-restart-elasticsearch/270665/5 "2021-05-18T10:10:09Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
