# ES cluster seems to be crashing in middle of a snapshot creation and then return not found errors when queried with the API wrapper

**URL:** <https://discuss.elastic.co/t/es-cluster-seems-to-be-crashing-in-middle-of-a-snapshot-creation-and-then-return-not-found-errors-when-queried-with-the-api-wrapper/234005>\
**Category:** Elasticsearch\
**Tags:** slm-snapshot-lifecycle-management\
**Created:** [May 23, 2020, 3:53pm UTC](https://discuss.elastic.co/t/es-cluster-seems-to-be-crashing-in-middle-of-a-snapshot-creation-and-then-return-not-found-errors-when-queried-with-the-api-wrapper/234005 "2020-05-23T15:53:18Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![ishanjain](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ishanjain/32/59637_2.png) [@ishanjain](https://discuss.elastic.co/u/ishanjain)\
**Post date:** [May 23, 2020, 3:53pm UTC](https://discuss.elastic.co/t/es-cluster-seems-to-be-crashing-in-middle-of-a-snapshot-creation-and-then-return-not-found-errors-when-queried-with-the-api-wrapper/234005/1 "2020-05-23T15:53:19Z")

</div>

Hello!

We have a Elasticsearch cluster running in production. We are using [github.com/olivere/elastic(V7)](http://github.com/olivere/elastic(V7)) to create snapshots.

Here is how it goes,

1. Create a `*elastic.Client`, Verify authentication works by fetching the ES version.

2. Trigger a snapshot creation in async mode. Then, Every 1 minute we check the status of snapshot creation.

Here is what the logs looks like,

```auto
time="2020-05-17T03:36:12Z" level=warning msg="error in fetching snapshot state. Try 1 of 5: elastic: Error 503 (Service Unavailable)" repository=esbackup-mw-elk-prod snapshot=20200516032609
time="2020-05-18T03:32:12Z" level=warning msg="error in fetching snapshot state. Try 2 of 5: elastic: Error 429 (Too Many Requests): [parent] Data too large, data for [<http_request>] would be [8162093384/7.6gb], which is larger than the limit of [8127315968/7.5gb], real usage: [8162093384/7.6gb], new bytes reserved: [0/0b] [type=circuit_breaking_exception]" repository=esbackup-mw-elk-prod snapshot=20200516032609
time="2020-05-18T03:40:12Z" level=warning msg="error in fetching snapshot state. Try 3 of 5: elastic: Error 503 (Service Unavailable)" repository=esbackup-mw-elk-prod snapshot=20200516032609
time="2020-05-20T10:15:09Z" level=warning msg="error in fetching snapshot state. Try 4 of 5: elastic: Error 404 (Not Found): [esbackup-mw-elk-prod:20200516032609] is missing [type=snapshot_missing_exception]" repository=esbackup-mw-elk-prod snapshot=20200516032609

```

Upon googling, I learned, circuit\_breaking\_exception can occur if there is less memory than needed to complete a operation[1].

What I don't understand is, Why it starts returning "Not Found" after a while and basically never recovers.

If I send a curl request to get snapshot state, It reports just fine. If i use the elastic wrapper, It keeps returning Not Found errors.

[1] [https://stackoverflow.com/questions/37216300/elasticsearch-circuit-breaking-exception-data-too-large-with-significant-terms](https://stackoverflow.com/questions/37216300/elasticsearch-circuit-breaking-exception-data-too-large-with-significant-terms)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [June 20, 2020, 3:53pm UTC](https://discuss.elastic.co/t/es-cluster-seems-to-be-crashing-in-middle-of-a-snapshot-creation-and-then-return-not-found-errors-when-queried-with-the-api-wrapper/234005/2 "2020-06-20T15:53:24Z")

</div>

Closed on June 20, 2020, 3:53pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.