# TransportError(503, 'cluster\_block\_exception', 'blocked by: \[SERVICE\_UNAVAILABLE/2/no master\];')

**URL:** <https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247>\
**Category:** Elastic Cloud on Kubernetes (ECK)\
**Created:** [May 12, 2020, 2:45pm UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247 "2020-05-12T14:45:27Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![Cloud\_newbie](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@Cloud\_newbie](https://discuss.elastic.co/u/Cloud_newbie)\
**Post date:** [May 12, 2020, 2:45pm UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247/1 "2020-05-12T14:45:28Z")

</div>

Hello,

I have deployed an ECK on GKE following the [quickstart](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-quickstart.html) documentation and 3 nodes.

The ES YAML file looks like this:

```auto
 cat <<EOF | kubectl apply -f -
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: quickstart
spec:
  version: 7.6.2
  nodeSets:
  -name: default
    count: 3
    config:
      node.master: true
      node.data: true
      node.ingest: true
      node.store.allow_mmap: false
EOF

```

And Kibana:

```auto
cat <<EOF | kubectl apply -f -
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: quickstart
spec:
  version: 7.6.2
  count: 1
  elasticsearchRef:
    name: quickstart
EOF

```

The GKE has 300 GBs of storage and each node 8 GBs RAM. When I am trying to upload a JSON of size 13 MBs using the Python API it works until the 2.4% and then it complains about:

"TransportError(503, 'cluster\_block\_exception', 'blocked by: [SERVICE\_UNAVAILABLE/2/no master];')"

And when I visit [https://ext-kb-IP:5600](https://ext-kb-IP:5600) I get:  
"Kibana server is not ready yet"

```auto
kubectl get svc
kubernetes ClusterIP 10.48.0.1 <none> 443/TCP 16m
quickstart-es-default ClusterIP None <none> <none> 12m
quickstart-es-http LoadBalancer 10.48.10.166 ext-IP 9200:31625/TCP 12m
quickstart-es-transport ClusterIP None <none> 9300/TCP 12m
quickstart-kb-http LoadBalancer 10.48.12.76 ext-kb-IP 5601:30168/TCP 9m6s

```

```auto
kubectl get pods
NAME READY STATUS RESTARTS AGE
quickstart-es-default-0 1/1 Running 0 111m
quickstart-es-default-1 1/1 Running 0 111m
quickstart-es-default-2 1/1 Running 0 111m
quickstart-kb-655d9b679c-cbp8l 1/1 Running 0 28m

```

```auto
curl -XGET -u "elastic:PASSWORD" -k "https://localhost:9200/_cluster/health?pretty":
    {
      "error" : {
        "root_cause" : [
          {
            "type" : "master_not_discovered_exception",
            "reason" : null
          }
        ],
        "type" : "master_not_discovered_exception",
        "reason" : null
      },
      "status" : 503
    }

```

The kibana pod logs:

```auto
{"type":"log","@timestamp":"2020-05-12T13:49:28Z","tags":["info","plugins-service"],"pid":6,"message":"Plugin \"case\" is disabled."}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins-system"],"pid":6,"message":"Setting up [37] plugins: [taskManager,siem,licensing,infra,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,reporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","taskManager"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","siem"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","licensing"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","infra"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","encryptedSavedObjects"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":6,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","code"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","usageCollection"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","metrics"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","canvas"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","timelion"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","features"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","security"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","apm_oss"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","translations"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","data"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","share"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","home"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","spaces"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","cloud"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","apm"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","graph"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","plugins","bfetch"],"pid":6,"message":"Setting up plugin"}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","savedobjects-service"],"pid":6,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
{"type":"log","@timestamp":"2020-05-12T13:50:00Z","tags":["info","savedobjects-service"],"pid":6,"message":"Starting saved objects migrations"}
Could not create APM Agent configuration: Request Timeout after 30000ms
{"type":"log","@timestamp":"2020-05-12T13:50:30Z","tags":["warning","savedobjects-service"],"pid":6,"message":"Unable to connect to Elasticsearch. Error: Request Timeout after 30000ms"}

```

---

<div class="post-metadata">

**Author:** ![charith-elastic](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@charith-elastic](https://discuss.elastic.co/u/charith-elastic)\
**Post date:** [May 12, 2020, 3:32pm UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247/2 "2020-05-12T15:32:42Z")

</div>

Do you see any errors in Elasticsearch logs? You can get a quick overview of all Elasticsearch nodes by running:

```auto
kubectl logs -l elasticsearch.k8s.elastic.co/cluster-name=quickstart --prefix --ignore-errors

```

Alternatively, drill down to each node individually by running (replace `quickstart-es-default-0` with the desired node name):

```auto
kubectl logs -f quickstart-es-default-0

```

---

<div class="post-metadata">

**Author:** ![Cloud\_newbie](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@Cloud\_newbie](https://discuss.elastic.co/u/Cloud_newbie)\
**Post date:** [May 12, 2020, 10:09pm UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247/3 "2020-05-12T22:09:22Z")

</div>

Thank you @charith-elastic for your immediate response 🙂

These are my logs:

```auto
[pod/quickstart-es-default-0/elasticsearch] "at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) [x-pack-security-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-0/elasticsearch] "at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) [x-pack-security-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-0/elasticsearch] "at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) [x-pack-security-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-0/elasticsearch] "at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-0/elasticsearch] "at org.elasticsearch.transport.TransportService$7.doRun(TransportService.java:762) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-0/elasticsearch] "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-0/elasticsearch] "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-0/elasticsearch] "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
[pod/quickstart-es-default-0/elasticsearch] "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
[pod/quickstart-es-default-0/elasticsearch] "at java.lang.Thread.run(Thread.java:830) [?:?]"] }
[pod/quickstart-es-default-1/elasticsearch] "stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
[pod/quickstart-es-default-1/elasticsearch] "at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-1/elasticsearch] "at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-1/elasticsearch] "at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-1/elasticsearch] "at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-1/elasticsearch] "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-1/elasticsearch] "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
[pod/quickstart-es-default-1/elasticsearch] "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
[pod/quickstart-es-default-1/elasticsearch] "at java.lang.Thread.run(Thread.java:830) [?:?]"] }
[pod/quickstart-es-default-1/elasticsearch] {"type": "server", "timestamp": "2020-05-12T22:06:13,596Z", "level": "DEBUG", "component": "o.e.a.s.m.TransportMasterNodeAction", "cluster.name": "quickstart", "node.name": "quickstart-es-default-1", "message": "no known master node, scheduling a retry", "cluster.uuid": "RMrnVfjKTju77enD-oZGhQ", "node.id": "DZFHEZ4SQqmAA8FwUeVV_g" }
[pod/quickstart-es-default-2/elasticsearch] "stacktrace": ["org.elasticsearch.discovery.MasterNotDiscoveredException: null",
[pod/quickstart-es-default-2/elasticsearch] "at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction$2.onTimeout(TransportMasterNodeAction.java:220) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-2/elasticsearch] "at org.elasticsearch.cluster.ClusterStateObserver$ContextPreservingListener.onTimeout(ClusterStateObserver.java:325) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-2/elasticsearch] "at org.elasticsearch.cluster.ClusterStateObserver$ObserverClusterStateListener.onTimeout(ClusterStateObserver.java:252) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-2/elasticsearch] "at org.elasticsearch.cluster.service.ClusterApplierService$NotifyTimeout.run(ClusterApplierService.java:598) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-2/elasticsearch] "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:633) [elasticsearch-7.6.2.jar:7.6.2]",
[pod/quickstart-es-default-2/elasticsearch] "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]",
[pod/quickstart-es-default-2/elasticsearch] "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]",
[pod/quickstart-es-default-2/elasticsearch] "at java.lang.Thread.run(Thread.java:830) [?:?]"] }
[pod/quickstart-es-default-2/elasticsearch] {"type": "server", "timestamp": "2020-05-12T22:06:06,403Z", "level": "WARN", "component": "o.e.c.c.ClusterFormationFailureHelper", "cluster.name": "quickstart", "node.name": "quickstart-es-default-2", "message": "master not discovered or elected yet, an election requires at least 2 nodes with ids from [DZFHEZ4SQqmAA8FwUeVV_g, DbeD2hHzSvqR7Yw9lyvHbw, zyqwPg1STFGjc0ee3DNUKw], have discovered [{quickstart-es-default-2}{DbeD2hHzSvqR7Yw9lyvHbw}{utXW_-IfRAKEhM3-q83E2Q}{10.44.2.8}{10.44.2.8:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, {quickstart-es-default-1}{DZFHEZ4SQqmAA8FwUeVV_g}{x3PxPaP2Ru2f-HgO2go8SA}{10.44.0.5}{10.44.0.5:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}, {quickstart-es-default-0}{zyqwPg1STFGjc0ee3DNUKw}{1ZTrBr7qQ3iptnK26j8uvA}{10.44.1.6}{10.44.1.6:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] which is a quorum; discovery will continue using [127.0.0.1:9300, 127.0.0.1:9301, 127.0.0.1:9302, 127.0.0.1:9303, 127.0.0.1:9304, 127.0.0.1:9305, 10.44.0.5:9300, 10.44.1.6:9300] from hosts providers and [{quickstart-es-default-2}{DbeD2hHzSvqR7Yw9lyvHbw}{utXW_-IfRAKEhM3-q83E2Q}{10.44.2.8}{10.44.2.8:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, {quickstart-es-default-1}{DZFHEZ4SQqmAA8FwUeVV_g}{x3PxPaP2Ru2f-HgO2go8SA}{10.44.0.5}{10.44.0.5:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}, {quickstart-es-default-0}{zyqwPg1STFGjc0ee3DNUKw}{1ZTrBr7qQ3iptnK26j8uvA}{10.44.1.6}{10.44.1.6:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}] from last-known cluster state; node term 3, last-accepted version 65 in term 3", "cluster.uuid": "RMrnVfjKTju77enD-oZGhQ", "node.id": "DbeD2hHzSvqR7Yw9lyvHbw" }
```

---

<div class="post-metadata">

**Author:** ![charith-elastic](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@charith-elastic](https://discuss.elastic.co/u/charith-elastic)\
**Post date:** [May 13, 2020, 7:25am UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247/4 "2020-05-13T07:25:42Z")

</div>

What seems like an interesting stacktrace on `quickstart-es-default-0` node is cutoff in the output. Please provide the full logs from `quickstart-es-default-0`.

---

<div class="post-metadata">

**Author:** ![Cloud\_newbie](https://avatars.discourse-cdn.com/v4/letter/c/ac8455/32.png) [@Cloud\_newbie](https://discuss.elastic.co/u/Cloud_newbie)\
**Post date:** [May 13, 2020, 8:48am UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247/5 "2020-05-13T08:48:09Z")

</div>

```auto
"at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1586) ~[lucene-core-8.4.0.jar:8.4.0 bc02ab906445fcf4e297f4ef00ab4a54fdd72ca2 - jpountz - 2019-12-19 20:16:14]",
"at org.elasticsearch.gateway.PersistedClusterStateService$MetaDataIndexWriter.updateGlobalMetaData(PersistedClusterStateService.java:506) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.gateway.PersistedClusterStateService$Writer.addMetaData(PersistedClusterStateService.java:711) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.gateway.PersistedClusterStateService$Writer.overwriteMetaData(PersistedClusterStateService.java:702) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.gateway.PersistedClusterStateService$Writer.writeFullStateAndCommit(PersistedClusterStateService.java:587) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState.setCurrentTerm(GatewayMetaState.java:511) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:198) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:453) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.ensureTermAtLeast(Coordinator.java:445) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:966) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at java.util.Optional.ifPresent(Optional.java:176) ~[?:?]",
"at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:525) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:491) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:123) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:370) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:354) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:378) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:186) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) ~[?:?]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
"at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
"at java.lang.Thread.run(Thread.java:830) ~[?:?]"] }
{"type": "server", "timestamp": "2020-05-13T08:37:52,986Z", "level": "INFO", "component": "o.e.c.c.JoinHelper", "cluster.name": "quickstart", "node.name": "quickstart-es-default-0", "message": "failed to join {quickstart-es-default-1}{DZFHEZ4SQqmAA8FwUeVV_g}{x3PxPaP2Ru2f-HgO2go8SA}{10.44.0.5}{10.44.0.5:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true} with JoinRequest{sourceNode={quickstart-es-default-0}{zyqwPg1STFGjc0ee3DNUKw}{1ZTrBr7qQ3iptnK26j8uvA}{10.44.1.6}{10.44.1.6:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, optionalJoin=Optional[Join{term=20819, lastAcceptedTerm=3, lastAcceptedVersion=69, sourceNode={quickstart-es-default-0}{zyqwPg1STFGjc0ee3DNUKw}{1ZTrBr7qQ3iptnK26j8uvA}{10.44.1.6}{10.44.1.6:9300}{dilm}{ml.machine_memory=2147483648, xpack.installed=true, ml.max_open_jobs=20}, targetNode={quickstart-es-default-1}{DZFHEZ4SQqmAA8FwUeVV_g}{x3PxPaP2Ru2f-HgO2go8SA}{10.44.0.5}{10.44.0.5:9300}{dilm}{ml.machine_memory=2147483648, ml.max_open_jobs=20, xpack.installed=true}}]}", "cluster.uuid": "RMrnVfjKTju77enD-oZGhQ", "node.id": "zyqwPg1STFGjc0ee3DNUKw" , 
"stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [quickstart-es-default-1][10.44.0.5:9300][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: No space left on device",
"at org.elasticsearch.ExceptionsHelper.convertToRuntime(ExceptionsHelper.java:59) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState.handleExceptionOnWrite(GatewayMetaState.java:571) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.gateway.GatewayMetaState$LucenePersistedState.setCurrentTerm(GatewayMetaState.java:517) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.CoordinationState.handleStartJoin(CoordinationState.java:198) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.joinLeaderInTerm(Coordinator.java:453) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.ensureTermAtLeast(Coordinator.java:445) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.handleJoin(Coordinator.java:966) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at java.util.Optional.ifPresent(Optional.java:176) ~[?:?]",
"at org.elasticsearch.cluster.coordination.Coordinator.processJoinRequest(Coordinator.java:525) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.lambda$handleJoinRequest$7(Coordinator.java:491) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.ConnectionManager.connectToNode(ConnectionManager.java:123) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:370) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.TransportService.connectToNode(TransportService.java:354) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.Coordinator.handleJoinRequest(Coordinator.java:478) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.cluster.coordination.JoinHelper.lambda$new$0(JoinHelper.java:125) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler$1.doRun(SecurityServerTransportInterceptor.java:257) ~[?:?]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:225) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.lambda$messageReceived$0(SecurityServerTransportInterceptor.java:306) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorizeSystemUser(AuthorizationService.java:378) ~[?:?]",
"at org.elasticsearch.xpack.security.authz.AuthorizationService.authorize(AuthorizationService.java:186) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.lambda$inbound$1(ServerTransportFilter.java:130) ~[?:?]",
"at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$authenticateAsync$2(AuthenticationService.java:248) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$lookForExistingAuthentication$6(AuthenticationService.java:310) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lookForExistingAuthentication(AuthenticationService.java:321) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.authenticateAsync(AuthenticationService.java:245) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.access$000(AuthenticationService.java:196) ~[?:?]",
"at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:139) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.ServerTransportFilter$NodeProfile.inbound(ServerTransportFilter.java:121) ~[?:?]",
"at org.elasticsearch.xpack.security.transport.SecurityServerTransportInterceptor$ProfileSecuredRequestHandler.messageReceived(SecurityServerTransportInterceptor.java:313) ~[?:?]",
"at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.transport.InboundHandler$RequestHandler.doRun(InboundHandler.java:264) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:692) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.6.2.jar:7.6.2]",
"at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
"at java.lang.Thread.run(Thread.java:830) [?:?]",
"Caused by: java.io.IOException: No space left on device",

```

---

<div class="post-metadata">

**Author:** ![charith-elastic](https://avatars.discourse-cdn.com/v4/letter/c/3ec8ea/32.png) [@charith-elastic](https://discuss.elastic.co/u/charith-elastic)\
**Post date:** [May 13, 2020, 10:12am UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247/6 "2020-05-13T10:12:03Z")

</div>

```auto
[quickstart-es-default-1][10.44.0.5:9300][internal:cluster/coordination/join]",
"Caused by: org.elasticsearch.ElasticsearchException: java.io.IOException: No space left on device"

```

It looks like your `quickstart-es-default-1` node has run out of disk space. You can add another `nodeSet` (or rename the existing one in the manifest) with a larger `volumeClaimTemplate` to request enough storage to suit your needs. See: [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html)

Example of renaming a `nodeSet` to add more space: [https://github.com/elastic/cloud-on-k8s/issues/325#issue-404254925](https://github.com/elastic/cloud-on-k8s/issues/325#issue-404254925)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 4, 2022, 7:57am UTC](https://discuss.elastic.co/t/transporterror-503-cluster-block-exception-blocked-by-service-unavailable-2-no-master/232247/7 "2022-11-04T07:57:53Z")

</div>