# Elasticsearch stops in minutes when restarted **URL:** \ **Category:** Elasticsearch\ **Created:** [September 5, 2019, 10:22am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221 "2019-09-05T10:22:24Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 5, 2019, 10:22am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/1 "2019-09-05T10:22:25Z")
When I restart Elasticsearch, it stops more than 10 minutes at "logger":"o.e.g.GatewayService", "log":"recovered [0] indices into cluster\_state" I don't know the root cause. Please help me. Thanks for your time! ``` {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:16.669Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[es_node] ","log":"loaded module [transport-netty4]"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:16.669Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[es_node] ","log":"loaded module [tribe]"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:16.669Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[es_node] ","log":"loaded plugin [ingest-attachment]"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:16.670Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[es_node] ","log":"loaded plugin [ingest-geoip]"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:16.670Z","logger":"o.e.p.PluginsService","timezone":"UTC","marker":"[es_node] ","log":"loaded plugin [ingest-user-agent]"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"WARN","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:19.480Z","logger":"o.e.d.c.s.Settings","timezone":"UTC","marker":"[es_node] ","log":"[http.enabled] setting was deprecated in Elasticsearch and will be removed in a future release! See the breaking changes documentation for the next major version."} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:20.487Z","logger":"o.e.d.DiscoveryModule","timezone":"UTC","marker":"[es_node] ","log":"using discovery type [zen] and host providers [settings]"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:21.204Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[es_node] ","log":"initialized"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:21.204Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[es_node] ","log":"starting ..."} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:21.357Z","logger":"o.e.t.TransportService","timezone":"UTC","marker":"[es_node] ","log":"publish_address {172.17.0.15:9300}, bound_addresses {0.0.0.0:9300}"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:21.371Z","logger":"o.e.b.BootstrapChecks","timezone":"UTC","marker":"[es_node] ","log":"bound or publishing to a non-loopback address, enforcing bootstrap checks"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:24.444Z","logger":"o.e.c.s.MasterService","timezone":"UTC","marker":"[es_node] ","log":"zen-disco-elected-as-master ([0] nodes joined), reason: new_master {es_node}{B7xQoWLSQ2KV5vYUrJFRlA}{YwMOe7tQRu6qbh9xRZsCfg}{172.17.0.15}{172.17.0.15:9300}"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:24.454Z","logger":"o.e.c.s.ClusterApplierService","timezone":"UTC","marker":"[es_node] ","log":"new_master {es_node}{B7xQoWLSQ2KV5vYUrJFRlA}{YwMOe7tQRu6qbh9xRZsCfg}{172.17.0.15}{172.17.0.15:9300}, reason: apply cluster state (from master [master {es_node}{B7xQoWLSQ2KV5vYUrJFRlA}{YwMOe7tQRu6qbh9xRZsCfg}{172.17.0.15}{172.17.0.15:9300} committed version [1] source [zen-disco-elected-as-master ([0] nodes joined)]])"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:24.488Z","logger":"o.e.h.n.Netty4HttpServerTransport","timezone":"UTC","marker":"[es_node] ","log":"publish_address {172.17.0.15:9200}, bound_addresses {0.0.0.0:9200}"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:33:24.488Z","logger":"o.e.n.Node","timezone":"UTC","marker":"[es_node] ","log":"started"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T **09:33:24**.500Z","logger":"o.e.g.GatewayService","timezone":"UTC","marker":"[es_node] ","log":"recovered [0] indices into cluster_state"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"WARN","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T **09:48:36**.955Z","logger":"o.e.d.c.m.MetaDataCreateIndexService","timezone":"UTC","marker":"[es_node] ","log":"the default number of shards will change from [5] to [1] in 7.0.0; if you wish to continue using the default of [5] shards, you must manage this on the create index request or with an index template"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:48:37.046Z","logger":"o.e.c.m.MetaDataCreateIndexService","timezone":"UTC","marker":"[es_node] ","log":"[logstash-2019.09.05] creating index, cause [auto(bulk api)], templates [], shards [5]/[1], mappings []"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:48:37.539Z","logger":"o.e.c.m.MetaDataMappingService","timezone":"UTC","marker":"[es_node] ","log":"[logstash-2019.09.05/Ie4GTU_UQ0KGvGYH2qdsag] create_mapping [fluentd]"} {"type":"log","host":"local-altiplano-elasticsearch-0","level":"INFO","systemid":"e7c84e34d38a49e1ae639a5dab455af5","system":"BELK","time": "2019-09-05T09:48:38.350Z","logger":"o.e.c.m.MetaDataMappingService","timezone":"UTC","marker":"[es_node] ","log":"[logstash-2019.09.05/Ie4GTU_UQ0KGvGYH2qdsag] update_mapping [fluentd]"} ``` ---
**Author:** ![msarwar](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@msarwar](https://discuss.elastic.co/u/msarwar)\ **Post date:** [September 6, 2019, 2:24am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/2 "2019-09-06T02:24:06Z")
Can you share your elasticsearch.yml please? And any other settings you are using? ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 6, 2019, 2:34am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/3 "2019-09-06T02:34:47Z")
> [@msarwar](#): > > an you share your elasticsearch.yml please? And any other settings you are using? The Elasticsearch.yml is the following: version: '3.4' networks: esnet: external: true services: kibana: image: "${ARTIFACTORY\_IP}:9000/elk\_k:${KIBANA\_VERSION}" container\_name: anv-kibana restart: always ports: - "5601:5601" environment: - ELASTICSEARCH\_URL=[http://elasticsearch:9200](http://elasticsearch:9200) - TZ=UTC depends\_on: - elasticsearch networks: - esnet elasticsearch: image: "${ARTIFACTORY\_IP}:9000/elk\_e:${ES\_VERSION}" container\_name: anv-elasticsearch restart: always ports: - "9200:9200" volumes: - /fnms/data\_backup:/data/elasticsearch\_backup - /fnms/config:/etc/elasticsearch/config environment: - ES\_JAVA\_OPTS=-Xms26g -Xmx26g - TZ=UTC networks: - esnet The .env is the following: ARTIFACTORY\_IP=10.131.69.114 KIBANA\_VERSION=6.5.4-19.03.01 ES\_VERSION=6.5.4-19.01.0 The issue only occurs when restarting Elasticsearch. Please advise. Thank you! ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 6, 2019, 2:54am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/4 "2019-09-06T02:54:26Z")
Actually, the issue doesn't always appear, it just usually occurs. ---
**Author:** ![msarwar](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@msarwar](https://discuss.elastic.co/u/msarwar)\ **Post date:** [September 6, 2019, 6:34am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/5 "2019-09-06T06:34:56Z")
Given that you don't make any changes to your config and If the error is not consistent then its hard to get down to the issue. You can turn on the DEBUG log to see the debug output. Also make sure to monitor disk and memory resources. ---
**Author:** ![msarwar](https://avatars.discourse-cdn.com/v4/letter/m/13edae/32.png) [@msarwar](https://discuss.elastic.co/u/msarwar)\ **Post date:** [September 6, 2019, 6:37am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/6 "2019-09-06T06:37:53Z")
> [@lmqnam](#): > > version: '3.4' > networks: > esnet: > external: true > services: > kibana: > image: "{ARTIFACTORY\_IP}:9000/elk\_k:{KIBANA\_VERSION}" > container\_name: anv-kibana > restart: always > ports: > > - "5601:5601" > environment: > - ELASTICSEARCH\_URL=[http://elasticsearch:9200](http://elasticsearch:9200) > - TZ=UTC > depends\_on: > - elasticsearch > networks: > - esnet > elasticsearch: > image: "{ARTIFACTORY\_IP}:9000/elk\_e:{ES\_VERSION}" > container\_name: anv-elasticsearch > restart: always > ports: > - "9200:9200" > volumes: > - /fnms/data\_backup:/data/elasticsearch\_backup > - /fnms/config:/etc/elasticsearch/config > environment: > - ES\_JAVA\_OPTS=-Xms26g -Xmx26g > - TZ=UTC > networks: > - esnet This is your docker file not elasticsearch.yml ---
**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Post date:** [September 6, 2019, 7:23am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/7 "2019-09-06T07:23:41Z")
The log posted tells you Elasticsearch is started and running. What do you get with curl -XGET [http://elasticsearch:9200/](http://elasticsearch:9200/) ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 6, 2019, 7:31am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/8 "2019-09-06T07:31:32Z")
Yes, it's running. But it stops at o.e.g.GatewayService more than 10 minutes. This only occurs when it's restarted. ---
**Author:** ![pjanzen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pjanzen/32/13756_2.png) [@pjanzen](https://discuss.elastic.co/u/pjanzen)\ **Post date:** [September 6, 2019, 7:34am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/9 "2019-09-06T07:34:32Z")
And when that happens, there is nothing else in the logs? Are you able to logon to the machine and do a journal -xe or service elasticsearch status I am not really familiar with docker images so that might not be possible... ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 6, 2019, 11:09am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/10 "2019-09-06T11:09:58Z")
> [@pjanzen](#): > > Are you able to logon Actually, when Elasticsearch is restarted, Kibana stops displaying logs. Then, Elasticsearch runs again and stops showing logs at o.e.g.GatewayService. Kibana only displays logs once Elasticsearch continues running MetaDataCreateIndexService and MetaDataMappingService. It takes more than 10 minutes for delaying. I don't know how to find the root cause. ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 6, 2019, 11:15am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/11 "2019-09-06T11:15:14Z")
Sorry. It's my mistakes. I uses docker to run Elasticsearch. I think the image below is elasticsearch.yml ![yml](https://us1.discourse-cdn.com/elastic/original/3X/8/8/8816dff1984954b473d637632e39d4d0322595c6.png) ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 6, 2019, 11:23am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/12 "2019-09-06T11:23:31Z")
Thanks for your advice. I tried, but it resets to info automatically when Elasticsearch is restarted. Do you have any ideas? ![properties](https://us1.discourse-cdn.com/elastic/original/3X/b/4/b411b3edaab3842ce5609e1d10d63568e6ee130c.png) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 6, 2019, 11:37am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/13 "2019-09-06T11:37:59Z")
> [@lmqnam](#): > > When I restart Elasticsearch, it stops more than 10 minutes at "logger":"o.e.g.GatewayService", "log":"recovered [0] indices into cluster\_state" I'm struggling to understand what the issue is here. This sounds like completely normal behaviour for Elasticsearch. It doesn't log much when it's running normally, and `recovered [0] indices into cluster_state` is usually the last line it logs when starting up. Can you explain more clearly why you think this is a problem? ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 9, 2019, 2:15am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/14 "2019-09-09T02:15:45Z")
> [@DavidTurner](#): > > it's running normally, and `recovered [0] indices into cluster_state` is usually the last line it logs when starting up. > > Can you explain more clearly why you think this is a problem? When it starts first, it runs and shows logs "logger":"o.e.c.m.MetaDataMappingService", "log":"[logstash-2019.09.05/Ie4GTU\_UQ0KGvGYH2qdsag] update\_mapping [fluentd]" as the last line without delay. But when restarted, it stops at "recovered [0] indices into cluster\_state" log in the period time and then continue to show "update\_mapping [fluentd]" log. Hence I think this is a problem. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [September 9, 2019, 5:14am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/15 "2019-09-09T05:14:34Z")
Are you using persistent storage for your data path `/data/data`? ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 9, 2019, 6:46am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/16 "2019-09-09T06:46:36Z")
I found out the root cause. It's due to the default 5s of request\_timeout parameter. The Fluentd process can get into this state when every attempt to write logs to an Elasticsearch instance takes longer than 5 seconds to complete. Until a consistent number of writes takes less than 5 seconds to complete, logging essentially stops working. In my case, if request\_timeout is set to 1m, it will be working fine. Thank you so much for your time! Refer: > **[uken/fluent-plugin-elasticsearch](https://github.com/uken/fluent-plugin-elasticsearch)** > > Contribute to uken/fluent-plugin-elasticsearch development by creating an account on GitHub. [https://bugzilla.redhat.com/show\_bug.cgi?id=1497836](https://bugzilla.redhat.com/show_bug.cgi?id=1497836) ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [September 9, 2019, 6:51am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/17 "2019-09-09T06:51:03Z")
Expanding on @Christian_Dahlqvist's question, the message indicates that there are no indices (i.e. no data at all) when the node restarts. It sounds like you previously inserted some data, so the question is why did it go away. The most likely explanation is that your data path is emptied when you restart your container. I don't think anything is wrong with Elasticsearch here (it's looks like it's working normally and waiting for data) but maybe the process that you expect to send it data is waiting to find the index it created before the node restarted; after 10 minutes perhaps it gives up and creates a new index. ---
**Author:** ![lmqnam](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/lmqnam/32/53624_2.png) [@lmqnam](https://discuss.elastic.co/u/lmqnam)\ **Post date:** [September 9, 2019, 6:56am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/18 "2019-09-09T06:56:07Z")
> [@DavidTurner](#): > > I don't think anything is wrong with Elasticsearch here (it's looks like it's working normally and waiting for data) but maybe the process that you expect to send it data is waiting to find the index it created before the node restarted; after 10 minutes perhaps it gives up and creates a new index. Yes, you're right. The root cause is the configuration for Fluentd. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 7, 2019, 7:02am UTC](https://discuss.elastic.co/t/elasticsearch-stops-in-minutes-when-restarted/198221/19 "2019-10-07T07:02:36Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.