# Elasticsearch restoration of a huge dump fails with client connection timeout errors

**URL:** <https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622>\
**Category:** Elasticsearch\
**Created:** [June 21, 2023, 7:37pm UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622 "2023-06-21T19:37:51Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![bhavaniprasad\_reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavaniprasad_reddy/32/52086_2.png) [@bhavaniprasad\_reddy](https://discuss.elastic.co/u/bhavaniprasad_reddy)\
**Post date:** [June 21, 2023, 7:37pm UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/1 "2023-06-21T19:37:51Z")

</div>

Hi Team,

I am trying to restore a **50GB** elastic dump file into a new elasticsearch cluster running with **3 elasticsearch replicas** on a **kubernetes cluster**.

The elasticsearch is running with a **7.10.2 oss** version image and using **managed-nfs-storage** as storage class. The restoration started without any issue and I started seeing the indices are in yellow state at the very beginning.

Later when I try to check the state of the indices then the API call fails with **client request timeout error** continuously.

**Error:**  
{"statusCode":502,"error":"Bad Gateway","message":"Client request timeout"}

I don't see any errors from the elasticsearch pod logs but found out that there are some **read write errors** noticed from the **nfs client** deployed on the worker nodes.

**NFS client errors** observed on **/var/log/messages** file of the worker nodes:

```auto
  Jun 21 14:57:48 cstream8-node kernel: NFS: __nfs4_reclaim_open_state: Lock reclaim failed!
  Jun 21 14:57:53 cstream8-node kernel: __nfs4_reclaim_open_state: 603 callbacks suppressed
  Jun 21 14:57:53 cstream8-node kernel: NFS: __nfs4_reclaim_open_state: Lock reclaim failed!

```

**Read Write errors** from **nfsiostat** command executed on worker nodes:

```auto
nfs-server:/mnt/k8sMount/elastic-index-persistent-storage-index-1-pvc-ece1d0d2-29c7-4298-9732-4ccee96d29ed mounted on /var/lib/kubelet/pods/19a68
986-2bc7-40ea-b439-1a234493d857/volumes/kubernetes.io~nfs/pvc-ece1d0d2-29c7-4298-9732-4ccee96d29ed:

           ops/s rpc bklog
         700.248 0.000

read: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) avg queue (ms) errors
                   5.752 710.568 123.532 0 (0.0%) 1.250 66.305 64.939 1763 (0.8%)
write: ops/s kB/s kB/op retrans avg RTT (ms) avg exe (ms) avg queue (ms) errors
                   1.961 999.801 509.876 0 (0.0%) 8.403 2147.932 2139.478 813 (1.0%)

```

I have 3 questions regarding the issue,

1. Is there any limitation on the datasize of the dump being restored?
2. Does elasticsearch support NFS as storage class?
3. Is there any way to restore such a huge dump without hitting any such errors?

**Details** :  
OS - Centos 8 stream  
Platform - RKE2 cluster  
NFS version - NFSV4

Please let me know if you need any more details.

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 21, 2023, 7:50pm UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/2 "2023-06-21T19:50:57Z")

</div>

> [@bhavaniprasad\_reddy](#):
>
> I am trying to restore a **50GB** elastic dump file into a new elasticsearch cluster running with **3 elasticsearch replicas** on a **kubernetes cluster**.

How are you restoring it ?

---

<div class="post-metadata">

**Author:** ![DineshNaik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dineshnaik/32/89308_2.png) [@DineshNaik](https://discuss.elastic.co/u/DineshNaik)\
**Post date:** [June 21, 2023, 8:00pm UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/3 "2023-06-21T20:00:38Z")

</div>

- Is there any limitation on the datasize of the dump being restored?  
--\> not sure on the upper limit , but 50GB is really small data set and you should be able to restore easily, based on how powerful your data nodes are .
- Does elasticsearch support NFS as storage class?  
--\> ECK does not come with its own storage mechanism for Elasticsearch data. It is compatible with any Kubernetes storage option. It is recommended to use PersistentVolumes, by configuring the [VolumeClaimTemplates](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-volume-claim-templates.html) section of the Elasticsearch resource.  
refer [https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-storage-recommendations.html](https://link) for more details.
- Is there any way to restore such a huge dump without hitting any such errors?  
--\> Are you trying to query the cluster where the restore is in progress?

---

<div class="post-metadata">

**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\
**Post date:** [June 21, 2023, 8:04pm UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/4 "2023-06-21T20:04:11Z")

</div>

> [@bhavaniprasad\_reddy](#):
>
> Is there any limitation on the datasize of the dump being restored?

Can you provide more context on what you are trying to restore and how? It is not clear.

Did you create a dump file using elasticdump or a similar tool? If yes, this is not an officially supported mode of restoring data to a Elasticsearch cluster, the official and recommended way to restore data into a cluster is using the Snapshot and Restore APIs.

> [@bhavaniprasad\_reddy](#):
>
> Does elasticsearch support NFS as storage class?

It supports, but it is not recommended to use NFS based for data paths, only for snapshots repositories.

> [@bhavaniprasad\_reddy](#):
>
> Is there any way to restore such a huge dump without hitting any such errors?

Need more context, as it is not clear what you are trying to restore and if this is a Elasticsearch issue or an infrastructure issue.

---

<div class="post-metadata">

**Author:** ![bhavaniprasad\_reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavaniprasad_reddy/32/52086_2.png) [@bhavaniprasad\_reddy](https://discuss.elastic.co/u/bhavaniprasad_reddy)\
**Post date:** [June 22, 2023, 6:19am UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/5 "2023-06-22T06:19:17Z")

</div>

- not sure on the upper limit , but 50GB is really small data set and you should be able to restore easily, based on how powerful your data nodes are .  
-\> can you please tell me how to check if the data nodes are really capable of handling this request?

- Are you trying to query the cluster where the restore is in progress?  
-\> Yes I am trying to list the indices while the restoration is in progress because the restoration is taking a very long time.

---

<div class="post-metadata">

**Author:** ![bhavaniprasad\_reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavaniprasad_reddy/32/52086_2.png) [@bhavaniprasad\_reddy](https://discuss.elastic.co/u/bhavaniprasad_reddy)\
**Post date:** [June 22, 2023, 6:29am UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/6 "2023-06-22T06:29:40Z")

</div>

- Did you create a dump file using elasticdump or a similar tool?  
-\> No, I am not using any tools for dump and restore instead I have used elastic APIs in order to take dump and restore the elastic dump,

```auto
PUT _snapshot/my_backup
{
    "type": "fs",
    "settings": {
    "location": "/usr/share/elasticsearch/backups",
    "compress": true
    }
}

POST _snapshot/my_backup/snapshot_1/_restore
{
  "indices" : "filebeat-*",
  "ignore_unavailable": true,
  "include_global_state": false,
  "include_aliases": false
}

```

- It supports, but it is not recommended to use NFS based for data paths, only for snapshots repositories.

- Need more context, as it is not clear what you are trying to restore and if this is a Elasticsearch issue or an infrastructure issue.  
-\> Even I too suspect the NFS storage since there are some read write errors appearing in the nfs statistics.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 22, 2023, 7:02am UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/7 "2023-06-22T07:02:28Z")

</div>

> [@bhavaniprasad\_reddy](#):
>
> The elasticsearch is running with a **7.10.2 oss** version image

This version is _very_ old, long past EOL. You must upgrade as a matter of urgency.

> [@bhavaniprasad\_reddy](#):
>
> {"statusCode":502,"error":"Bad Gateway","message":"Client request timeout"}

This error is not coming from Elasticsearch, it's a timeout imposed by something else. It does not mean that the restore has failed. I don't remember if anything changed in the few years since 7.10 was released, but in supported versions the restore will carry on in the background and eventually complete successfully.

> [@DineshNaik](#):
>
> It is compatible with any Kubernetes storage option.

> [@leandrojmp](#):
>
> It supports, but it is not recommended to use NFS based for data paths

It's a little more subtle than either of these statements, and I think [the manual describes it best](https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-node.html#data-path):

> Elasticsearch requires the filesystem to act as if it were backed by a local disk, but this means that it will work correctly on properly-configured remote block devices (e.g. a SAN) and remote filesystems (e.g. NFS) as long as the remote storage behaves no differently from local storage. [...] The performance of an Elasticsearch cluster is often limited by the performance of the underlying storage, so you must ensure that your storage supports acceptable performance. Some remote storage performs very poorly, especially under the kind of load that Elasticsearch imposes, so make sure to benchmark your system carefully before committing to a particular storage architecture.

i.e. you're free to use NFS if you want, but it's on you to make sure it is properly configured and performs adequately. If it doesn't, you need someone with NFS expertise to help you (and this is probably not the best forum to find such help).

---

<div class="post-metadata">

**Author:** ![bhavaniprasad\_reddy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bhavaniprasad_reddy/32/52086_2.png) [@bhavaniprasad\_reddy](https://discuss.elastic.co/u/bhavaniprasad_reddy)\
**Post date:** [June 22, 2023, 7:15am UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/8 "2023-06-22T07:15:52Z")

</div>

Thanks @DavidTurner for the details. Actually I am using 7.10.2 version because this is last version available in the OSS release.

I request your suggestion in using the latest versions of ELK since there are a lot of modifications made to the licensing part.

I have gone through the document but had a doubt in using the latest versions.

- Are they really free same as OSS?
- Are there any restrictions that I need to consider while using the latest versions of ELK?

Also I suspect that I might hit the same error with the latest versions of ELK if the issue is from NFS and not from ELK.

can you please share me any pointers in case the issue is with NFS client?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 22, 2023, 7:15am UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/9 "2023-06-22T07:15:52Z")

</div>

7.10.2 is [EOL](https://www.elastic.co/support/eol) and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [June 22, 2023, 7:21am UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/10 "2023-06-22T07:21:32Z")

</div>

> [@bhavaniprasad\_reddy](#):
>
> - Are they really free same as OSS?
> - Are there any restrictions that I need to consider while using the latest versions of ELK?

See [FAQ on 2021 License Change | Elastic](https://www.elastic.co/pricing/faq/licensing) for further details. Basically the only restriction is you cannot sell Elasticsearch-as-a-service.

> [@bhavaniprasad\_reddy](#):
>
> can you please share me any pointers in case the issue is with NFS client?

I have no further information beyond the docs link in my previous post. You will need to find some NFS experts to help you, and this isn't the right forum for that.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 20, 2023, 7:22am UTC](https://discuss.elastic.co/t/elasticsearch-restoration-of-a-huge-dump-fails-with-client-connection-timeout-errors/336622/11 "2023-07-20T07:22:04Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.