# Error 503 on server post unblocking the index on a proxy **URL:** \ **Category:** Elasticsearch\ **Created:** [December 5, 2020, 6:53am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720 "2020-12-05T06:53:14Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Anuradha12](https://avatars.discourse-cdn.com/v4/letter/a/ebca7d/32.png) [@Anuradha12](https://discuss.elastic.co/u/Anuradha12)\ **Post date:** [December 5, 2020, 6:53am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/1 "2020-12-05T06:53:14Z")
Could not save: {'index': {'\_index': 'prj\_1a9c0440af7f4ed297736cffa5591082\_connector', '\_type': '\_doc', '\_id': 'model\_d25q7003720200', 'status': 503, 'error': {'type': 'unavailable\_shards\_exception', 'reason': '[prj\_1a9c0440af7f4ed297736cffa5591082\_connector][4] primary shard is not active Timeout: [1m], request: [BulkShardRequest [[prj\_1a9c0440af7f4ed297736cffa5591082\_connector][4]] containing [402] requests]'}}} Need solution asap. thank you ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 5, 2020, 7:01am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/2 "2020-12-05T07:01:35Z")
Which version of Elasticsearch are you using? What is the full output of the following APIs? - [Cluster health API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/cluster-health.html) - [Cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/7.10/cluster-stats.html) ---
**Author:** ![Anuradha12](https://avatars.discourse-cdn.com/v4/letter/a/ebca7d/32.png) [@Anuradha12](https://discuss.elastic.co/u/Anuradha12)\ **Post date:** [December 5, 2020, 7:38am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/3 "2020-12-05T07:38:06Z")
{ "state" : "UNASSIGNED", "primary" : false, "node" : null, "relocating\_node" : null, "shard" : 3, "index" : "prj\_7269ebdb1b97fc9e9d63ba79692350ac\_assets", "recovery\_source" : { "type" : "PEER" }, "unassigned\_info" : { "reason" : "ALLOCATION\_FAILED", "at" : "2020-12-04T15:41:47.021Z", "failed\_attempts" : 5, "delayed" : false, "details" : "failed shard on node [QhG\_OTEpStehLVm8BKkzjw]: failed to create shard, failure AccessDeniedException[/usr/share/elasticsearch/data/nodes/0/indices/KmMVjuO7TqSec5cnapTspA/3]", "allocation\_status" : "no\_attempt" } }, { "state" : "UNASSIGNED", "primary" : false, "node" : null, "relocating\_node" : null, "shard" : 7, "index" : "prj\_7269ebdb1b97fc9e9d63ba79692350ac\_assets", "recovery\_source" : { "type" : "PEER" }, "unassigned\_info" : { "reason" : "ALLOCATION\_FAILED", "at" : "2020-12-04T15:41:46.812Z", "failed\_attempts" : 5, "delayed" : false, "details" : "failed shard on node [Yzmrlmh\_RVSulSr3j9a\_Ug]: failed to create shard, failure AccessDeniedException[/usr/share/elasticsearch/data/nodes/0/indices/KmMVjuO7TqSec5cnapTspA/7]", "allocation\_status" : "no\_attempt" } }, { "state" : "UNASSIGNED", "primary" : false, "node" : null, "relocating\_node" : null, "shard" : 2, "index" : "prj\_7269ebdb1b97fc9e9d63ba79692350ac\_assets", "recovery\_source" : { "type" : "PEER" }, "unassigned\_info" : { "reason" : "NODE\_LEFT", "at" : "2020-12-04T15:44:18.056Z", "delayed" : false, "details" : "node\_left [Yzmrlmh\_RVSulSr3j9a\_Ug]", "allocation\_status" : "no\_attempt" } }, This is the out put which i am getting and version is "number" : "6.8.0", ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 5, 2020, 7:39am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/4 "2020-12-05T07:39:18Z")
That is not the output of any of the APIs I asked for. Are you using shared storage for Elasticsearch? What is the specification of your cluster? ---
**Author:** ![Anuradha12](https://avatars.discourse-cdn.com/v4/letter/a/ebca7d/32.png) [@Anuradha12](https://discuss.elastic.co/u/Anuradha12)\ **Post date:** [December 5, 2020, 7:41am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/5 "2020-12-05T07:41:34Z")
Here you have it- curl -XGET "[http://127.0.0.1:9200/\_cluster/stats](http://127.0.0.1:9200/_cluster/stats)" {"\_nodes":{"total":2,"successful":2,"failed":0},"cluster\_name":"darksiders.cluster","cluster\_uuid":"2u0u2QbrS426ZyZqN\_m\_4w","timestamp":1607154046320,"status":"red","indices":{"count":5,"shards":{"total":12,"primaries":12,"replication":0.0,"index":{"shards":{"min":1,"max":4,"avg":2.4},"primaries":{"min":1,"max":4,"avg":2.4},"replication":{"min":0.0,"max":0.0,"avg":0.0}}},"docs":{"count":526516,"deleted":140228},"store":{"size\_in\_bytes":1886904179},"fielddata":{"memory\_size\_in\_bytes":0,"evictions":0},"query\_cache":{"memory\_size\_in\_bytes":9642744,"total\_count":11453,"hit\_count":4027,"miss\_count":7426,"cache\_size":565,"cache\_count":785,"evictions":220},"completion":{"size\_in\_bytes":0},"segments":{"count":101,"memory\_in\_bytes":11624034,"terms\_memory\_in\_bytes":9338581,"stored\_fields\_memory\_in\_bytes":136624,"term\_vectors\_memory\_in\_bytes":0,"norms\_memory\_in\_bytes":1480640,"points\_memory\_in\_bytes":404673,"doc\_values\_memory\_in\_bytes":263516,"index\_writer\_memory\_in\_bytes":0,"version\_map\_memory\_in\_bytes":0,"fixed\_bit\_set\_memory\_in\_bytes":87224,"max\_unsafe\_auto\_id\_timestamp":-1,"file\_sizes":{}}},"nodes":{"count":{"total":2,"data":1,"coordinating\_only":0,"master":1,"ingest":1},"versions":["6.8.0"],"os":{"available\_processors":48,"allocated\_processors":48,"names":[{"name":"Linux","count":2}],"pretty\_names":[{"pretty\_name":"CentOS Linux 7 (Core)","count":2}],"mem":{"total\_in\_bytes":134746841088,"free\_in\_bytes":44399255552,"used\_in\_bytes":90347585536,"free\_percent":33,"used\_percent":67}},"process":{"cpu":{"percent":0},"open\_file\_descriptors":{"min":493,"max":614,"avg":553}},"jvm":{"max\_uptime\_in\_millis":13477550653,"versions":[{"version":"12.0.1","vm\_name":"OpenJDK 64-Bit Server VM","vm\_version":"12.0.1+12","vm\_vendor":"Oracle Corporation","count":2}],"mem":{"heap\_used\_in\_bytes":2664205872,"heap\_max\_in\_bytes":4151836672},"threads":371},"fs":{"total\_in\_bytes":1181231001600,"free\_in\_bytes":824034258944,"available\_in\_bytes":764007268352},"plugins":,"network\_types":{"transport\_types":{"security4":2},"http\_types":{"security4":2}}}} And this: curl -XGET "[http://127.0.0.1:9200/\_cluster/health](http://127.0.0.1:9200/_cluster/health)" {"cluster\_name":"darksiders.cluster","status":"red","timed\_out":false,"number\_of\_nodes":2,"number\_of\_data\_nodes":1,"active\_primary\_shards":12,"active\_shards":12,"relocating\_shards":0,"initializing\_shards":0,"unassigned\_shards":168,"delayed\_unassigned\_shards":0,"number\_of\_pending\_tasks":0,"number\_of\_in\_flight\_fetch":0,"task\_max\_waiting\_in\_queue\_millis":0,"active\_shards\_percent\_as\_number":6.666666666666667} ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [December 5, 2020, 7:55am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/6 "2020-12-05T07:55:31Z")
> [@Christian\_Dahlqvist](#): > > Are you using shared storage for Elasticsearch? What is the specification of your cluster? It would help if you could tell us a bit more about the cluster and the hardware/storage used. > [@Anuradha12](#): > > "details" : "failed shard on node [Yzmrlmh\_RVSulSr3j9a\_Ug]: failed to create shard, failure AccessDeniedException[/usr/share/elasticsearch/data/nodes/0/indices/KmMVjuO7TqSec5cnapTspA/7]", This to me sounds like a potential file system problem. ---
**Author:** ![Anuradha12](https://avatars.discourse-cdn.com/v4/letter/a/ebca7d/32.png) [@Anuradha12](https://discuss.elastic.co/u/Anuradha12)\ **Post date:** [December 5, 2020, 8:03am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/7 "2020-12-05T08:03:20Z")
Hardware: CPU cores: 24 RAM:64Gb HDD: 1Tb and yes its a shared storage for ES , couchdb and postgres all this runs using docker container. ---
**Author:** ![Anuradha12](https://avatars.discourse-cdn.com/v4/letter/a/ebca7d/32.png) [@Anuradha12](https://discuss.elastic.co/u/Anuradha12)\ **Post date:** [December 5, 2020, 8:14am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/8 "2020-12-05T08:14:45Z")
There was no issue until 2 days back. If you can suggest anything to unblock this situation it would be really helpful. Yesterday one of the linked proxy had full disk storage so post increasing the space i just unblocked indexes using command - curl -XPUT -H "Content-Type: application/json" [http://localhost:9200/\_all/\_settings](http://localhost:9200/_all/_settings) -d '{"index.blocks.read\_only\_allow\_delete": null}' But this issue is happening on the server afterwards. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 2, 2021, 8:14am UTC](https://discuss.elastic.co/t/error-503-on-server-post-unblocking-the-index-on-a-proxy/257720/9 "2021-01-02T08:14:53Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.