# Elastic pod is not Ready: Readiness probe failed: nc: connect to 127.0.0.1 port 8080 (tcp) failed: Connection refused

**URL:** <https://discuss.elastic.co/t/elastic-pod-is-not-ready-readiness-probe-failed-nc-connect-to-127-0-0-1-port-8080-tcp-failed-connection-refused/368853>\
**Category:** Elasticsearch\
**Created:** [October 15, 2024, 2:26pm UTC](https://discuss.elastic.co/t/elastic-pod-is-not-ready-readiness-probe-failed-nc-connect-to-127-0-0-1-port-8080-tcp-failed-connection-refused/368853 "2024-10-15T14:26:05Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![nnikushkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nnikushkin/32/122950_2.png) [@nnikushkin](https://discuss.elastic.co/u/nnikushkin)\
**Post date:** [October 15, 2024, 2:26pm UTC](https://discuss.elastic.co/t/elastic-pod-is-not-ready-readiness-probe-failed-nc-connect-to-127-0-0-1-port-8080-tcp-failed-connection-refused/368853/1 "2024-10-15T14:26:05Z")

</div>

Hello everyone,

We had recently an upgrade of K8s nodes, during which one all pods were killed. Since the Elastic instances were deployed via ECK, the ECK operator automatically recreated the pods. However, the Elastic pod was marked as running but not ready:

```auto
# kubectl get pod elastic-main

NAME READY STATUS RESTARTS AGE
main-es-master-0 0/1 Running 0 5d20h

```

There were no errors in the pod logs, but the pod description showed the following error:

```auto
# kubectl describe pod elastic-main

Events:
  Warning Unhealthy 25s (x184 over 14m) kubelet Readiness probe failed: nc: connect to 127.0.0.1 port 8080 (tcp) failed: Connection refused

```

While the connection to Elastic on port 9200 was successful, the connection to port 8080 failed from inside the container:

```auto
# kubectl exec -it elastic-main -- bash

es@main-es-master-0:/usr/share/elasticsearch$ nc -z -v -w5 127.0.0.1 9200
Connection to 127.0.0.1 9200 port [tcp/*] succeeded!

es@main-es-master-0:/usr/share/elasticsearch$ nc -z -v -w5 127.0.0.1 8080
nc: connect to 127.0.0.1 port 8080 (tcp) failed: Connection refused

```

and indeed there was no service listening on port 8080 (while for correct Elastic there is):

```auto
es@main-es-master-0:/usr/share/elasticsearch$ cat /proc/net/tcp6 | grep " 0A " | awk '{print $2}' | cut -d: -f2 | xargs -I{} printf "%d\n" 0x{}

9200
9300

```

The cluster status was green though:

```auto
curl -u "admin:1qazXSW@" -k "https://localhost:9200/_cluster/health?filter_path=status,*_shards?pretty"
{"status":"green"}

```

There were no issues allocating shards:

```auto
curl -u "admin:1qazXSW@" -k "https://localhost:9200/_cluster/allocation/explain?pretty"
{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "No shard was specified in the request..."
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "No shard was specified in the request..."
  },
  "status" : 400
}

```

or problems with the disk space

I found that only removing Elastic along with its PVC and redeploying both from scratch resolves the issue.

I was able to reproduce the problem once by deploying Elastic, attempting to delete the Elastic pod, interrupting the deletion with CTRL+C, and then forcibly removing the Elastic pod with the `--force` flag. After this, ECK tries to recreate the pod but it failed with the following error:

```auto
{"@timestamp":"2024-10-15T10:15:50.036Z", "log.level":"DEBUG", "message":"address [10.10.10.10:9300], node [unknown discovery result", "ecs.version": "1.2.0","service.name":"ES_ECS","event.dataset":"elasticsearch.server","process.thread.name":"elasticsearch[feature-userstory-1733-es-master-0][generic][T#4]","log.logger":"org.elasticsearch.discovery.PeerFinder","elasticsearch.cluster.uuid":"PhXNclNITlux-nuW6ymzhA","elasticsearch.node.id":"hUKTwpuqSEiiZ8A8fWKUVA","elasticsearch.node.name":"feature-userstory-1733-es-master-0","elasticsearch.cluster.name":"feature-userstory-1733","error.type":"org.elasticsearch.transport.ConnectTransportException","error.message":"[][10.10.10.10:9300] connect_timeout[30s]","error.stack_trace":"org.elasticsearch.transport.ConnectTransportException: [][10.10.10.10:9300] connect_timeout[30s]\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.transport.TcpTransport$ChannelsConnectedListener.onTimeout(TcpTransport.java:1150)\n\tat org.elasticsearch.server@8.15.0/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:917)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)\n\tat java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)\n\tat java.base/java.lang.Thread.run(Thread.java:1570)\n"}

```

ECK tried the recreate it again and failed one more time

Finally, on the third attempt, it recreateed successfully, Elastic health was green, but a general "Readiness probe failed" error message remains in the pod description and the pod was marked as "Not Ready"

Currently, I have workarounded the issue with a custom readiness probe (though I know it is not recommended for versions \>8.2.0):

```auto
spec:
  containers:
  - name: elasticsearch
    readinessProbe:
      exec:
        command:
        - bash
        - -c
        - |
          curl -s -k https://localhost:9200 | grep -q "missing authentication credentials"
      failureThreshold: 3
      initialDelaySeconds: 10
      periodSeconds: 12
      successThreshold: 1
      timeoutSeconds: 12

```

which allowed me to check on 9200 instead of 8080

Summary:

- The Pod is marked as "Not Ready"
- The error message in the pod description is uninformative and does not clarify the misconfiguration.
- No errors were found in the logs, even with debug enabled.
- The pod is actually ready and responsive.

Details:

Elastic: 8.15.0  
ECK: 2.13.0  
Platform: Openshift

Any insights or suggestions on how to address this issue would be greatly appreciated!

Thank you!

---

<div class="post-metadata">

**Author:** ![Tomas\_Bahnik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/tomas_bahnik/32/138853_2.png) [@Tomas\_Bahnik](https://discuss.elastic.co/u/Tomas_Bahnik)\
**Post date:** [October 30, 2024, 11:17am UTC](https://discuss.elastic.co/t/elastic-pod-is-not-ready-readiness-probe-failed-nc-connect-to-127-0-0-1-port-8080-tcp-failed-connection-refused/368853/2 "2024-10-30T11:17:26Z")

</div>

I am facing similar issue but in my case the Elasticsearch readiness probe is set as  
readinessProbe:  
exec:  
command:  
- bash  
- -c  
- /mnt/elastic-internal/scripts/readiness-port-script.sh  
failureThreshold: 3  
initialDelaySeconds: 10  
periodSeconds: 5  
successThreshold: 1  
timeoutSeconds: 5

and it passes  
elasticsearch@elasticsearch-es-default-0:~$ /mnt/elastic-internal/scripts/readiness-port-script.sh  
Connection to 127.0.0.1 8080 port [tcp/\*] succeeded!

while the health is yellow even if the probe failed only 2x

Warning Unhealthy 32m (x2 over 32m) kubelet Readiness probe failed: nc: connect to 127.0.0.1 port 8080 (tcp) failed: Connection refused

version : 8.15.0 (eck-stack chart 0.12.1)

---

<div class="post-metadata">

**Author:** ![saidevops8989](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saidevops8989/32/141238_2.png) [@saidevops8989](https://discuss.elastic.co/u/saidevops8989)\
**Post date:** [February 7, 2025, 8:18pm UTC](https://discuss.elastic.co/t/elastic-pod-is-not-ready-readiness-probe-failed-nc-connect-to-127-0-0-1-port-8080-tcp-failed-connection-refused/368853/3 "2025-02-07T20:18:54Z")

</div>

Even i am facing same same issue  
I have deployed as elasticsearch with operator and having 3 pvc and 3 pods  
As you said if its new cluster its works fine but when i delete and recreation struck at  
Readiness probe failed: Elasticsearch is not ready yet. Check the server logs.

Even after my cluster health is green at pod

still is see Readiness probe failed: Elasticsearch is not ready yet. Check the server logs.  
As you mention no log  
By god grace thanks for the workaround it helped me @nnikushkin

---

<div class="post-metadata">

**Author:** ![saidevops8989](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/saidevops8989/32/141238_2.png) [@saidevops8989](https://discuss.elastic.co/u/saidevops8989)\
**Post date:** [April 3, 2025, 4:12pm UTC](https://discuss.elastic.co/t/elastic-pod-is-not-ready-readiness-probe-failed-nc-connect-to-127-0-0-1-port-8080-tcp-failed-connection-refused/368853/4 "2025-04-03T16:12:54Z")

</div>

Is there a long-term solution to this issue? Every time we reboot the nodes, the pods restart and my StatefulSet configuration seems to be overwritten, causing it to wait on port 8080 again.  
I have deployed with elastic operator how can i fix this permanently. @nnikushkin

---

<div class="post-metadata">

**Author:** ![nnikushkin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nnikushkin/32/122950_2.png) [@nnikushkin](https://discuss.elastic.co/u/nnikushkin)\
**Post date:** [April 4, 2025, 8:15am UTC](https://discuss.elastic.co/t/elastic-pod-is-not-ready-readiness-probe-failed-nc-connect-to-127-0-0-1-port-8080-tcp-failed-connection-refused/368853/5 "2025-04-04T08:15:55Z")

</div>

Hey!

I do not have a permanent fix for this, unfortunately

However, my fix was for my PRD Elastic instance deployed in Openshift via ECK operator and, since that day (Oct, 15 2024), they were nodes reload and ECK operator recreated Elastic pod without any problems. It did not overwrite Elastic rediness configuration to the default one, since the new configuration is explicitly mentioned in the container configuration section in the manifest file

The only option I see is diving into it together with Elastic Support further