# Elasticsearch throws 503 Server Unavailable error **URL:** \ **Category:** Elasticsearch\ **Created:** [December 8, 2023, 1:52pm UTC](https://discuss.elastic.co/t/elasticsearch-throws-503-server-unavailable-error/348898 "2023-12-08T13:52:51Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![Kalidastate](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kalidastate/32/129050_2.png) [@Kalidastate](https://discuss.elastic.co/u/Kalidastate)\ **Post date:** [December 8, 2023, 1:52pm UTC](https://discuss.elastic.co/t/elasticsearch-throws-503-server-unavailable-error/348898/1 "2023-12-08T13:52:51Z")
I am facing one issue with the Elasticsearch in the production environment. Elasticsearch stops responding to the API calls and it needs to be restarted. Logs collected from Elasticsearch are as follows When the issue occurs we start seeing following logs, after this all API calls fail ```auto [o.e.a.a.i.s.TransportIndicesStatsAction] failed to execute [indices:monitor/stats] on node [JD-iowTKQR-o_5hEjJSNow] org.elasticsearch.transport.ReceiveTimeoutTransportException: [GT-EVAULTIDX01][127.0.0.1:9201][indices:monitor/stats[n]] request_id [416253] timed out after [14847ms] [o.e.c.InternalClusterInfoService] failed to retrieve stats for node [JD-iowTKQR-o_5hEjJSNow] org.elasticsearch.transport.ReceiveTimeoutTransportException: [GT-EVAULTIDX01][127.0.0.1:9201][cluster:monitor/nodes/stats[n]] request_id [416252] timed out after [14847ms] ``` I am seeing some IO Exceptions, it says "An unexpected network error occurred". Does that mean the because of some network issue Elasticsearch is not working as expected? ```auto [o.e.m.f.FsHealthService] health check failed java.lang.IllegalStateException: environment is not locked at org.elasticsearch.env.NodeEnvironment.assertEnvIsLocked(NodeEnvironment.java:1106) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.env.NodeEnvironment.nodeDataPaths(NodeEnvironment.java:865) ~[elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.monitor.fs.FsHealthService$FsHealthMonitor.monitorFSHealth(FsHealthService.java:156) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.monitor.fs.FsHealthService$FsHealthMonitor.run(FsHealthService.java:144) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.threadpool.Scheduler$ReschedulingRunnable.doRun(Scheduler.java:214) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:777) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) [elasticsearch-7.16.3.jar:7.16.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] Caused by: java.io.IOException: An unexpected network error occurred ``` After this point Elasticsearch becomes unresponsive and it throws excetion to the API calls I am seeing following logs multiple time ```auto [r.suppressed] path: /_cat/nodes, params: {master_timeout=30s} org.elasticsearch.discovery.MasterNotDiscoveredException: null at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:297) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:345) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:263) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:660) [elasticsearch-7.16.3.jar:7.16.3] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:718) [elasticsearch-7.16.3.jar:7.16.3] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?] at java.lang.Thread.run(Thread.java:829) [?:?] [o.e.i.b.request] [request] Adjusted breaker by [16440] bytes, now [16440] [o.e.i.b.in_flight_requests] [in_flight_requests] Adjusted breaker by [0] bytes, now [0] [o.e.h.HttpTracer] [37512][null][SERVICE_UNAVAILABLE][application/json; charset=UTF-8][151] sent response to [Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:18391}] success [true] ``` Elasticsearch throws the 503 Server Unavailable error for any API call. ```auto http://localhost:9200/_cat/shards?v "error": { "root_cause": [ { "type": "master_not_discovered_exception", "reason": null } "type": "master_not_discovered_exception", "reason": null "status": 503 } ``` Currently, I not sure what is causing the problem to the Elasticsearch. Need suggestions how I can troubleshoot and resolve this issue ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 8, 2023, 2:37pm UTC](https://discuss.elastic.co/t/elasticsearch-throws-503-server-unavailable-error/348898/2 "2023-12-08T14:37:48Z")
Closed duplicate of > [@Elasticsearch throws error 503 Server Unavailable](https://discuss.elastic.co/t/elasticsearch-throws-error-503-server-unavailable/348896): > > I am facing one issue with the Elasticsearch in the production environment. Elasticsearch stops responding to the API calls and it needs to be restarted. Logs collected from Elasticsearch are as follows When the issue occurs we start seeing following logs, after this all API calls fail [o.e.a.a.i.s.TransportIndicesStatsAction] failed to execute [indices:monitor/stats] on node [JD-iowTKQR-o\_5hEjJSNow] org.elasticsearch.transport.ReceiveTimeoutTransportException: [GT-EVAULTIDX01][127.0.0.1:920… Please only open one topic at a time on the same subject. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [December 8, 2023, 2:37pm UTC](https://discuss.elastic.co/t/elasticsearch-throws-503-server-unavailable-error/348898/3 "2023-12-08T14:37:53Z")