Gateway timeout while getting the snapshots from Azure storage account

Elastic Stack Elasticsearch
1

Hi all, we are using Elastic Stack with Kibana, and we have created restore and backup option in Elastic UI, so we are connected to Azure storage account and we are doing hourly backup every day. Everything was fine until yesterday, usually when we click on all snapshots we need to wait little bit to see all snapshot, but from today we are gettin gateway timeout 504 error, it's like we have so many snapshots and Elastic UI can't get them from Azure Storage account and we get 504 error. Any help with this?

Thanks in advance

2

Hi there Milos,

This error isn't coming from Elasticsearch, it's some timeout imposed by something else in your environment. You'll need to make that timeout longer, or bypass whatever is timing out and make the request directly against Elasticsearch.

3

Hi David,

Thanks for information, but could be this something on client side on Elastic, I found this in documentation:

image
image1618×1252 325 KB

Thanks for support.

4

No, it's very unlikely you're hitting that timeout, and even if you were it wouldn't result in a 504.

5

After searching what could be the issue, we tried to create new policy for backup, and now we are hitting this error when policy try to create daily snapshot:

{
"type": "snapshot_exception",
"reason": "[backup:daily-snapshots-aysurmapt9mdlsur6--eaw/Rum4vqRDTEWvP_w63fd60Q] Indices don't have primary shards [.siem-signals-default-000035, .ds-.logs-deprecation.elasticsearch-default-2024.01.13-000037, .kibana-event-log-8.2.2-000020, .kibana-event-log-7.11.1-000034, .ds-ilm-history-5-2024.01.07-000053, .kibana-event-log-7.17.3-000020, .ds-.slm-history-5-2024.01.12-000036]"
}

Our elastic PVC is full, could this possible reson for that indices dont have signed primary shards?

Any advice, thanks in advance

6

See these docs for help with that.

7

After little bit a time this is what is happening, so in UI we get 504, and inside container we have errors logs and error in UI, is this related then some CPU or RAM issues on Elastic, I have checked that as well, all look fine.

image
image1667×1294 278 KB

2024-02-22T16:53:06+01:00 [2024-02-22T15:53:06.809+00:00][ERROR][http] TimeoutError: Request timed out
2024-02-22T16:53:06+01:00 at KibanaTransport.request (/usr/share/kibana/node_modules/@elastic/transport/lib/Transport.js:524:31)
2024-02-22T16:53:06+01:00 at runMicrotasks ()
2024-02-22T16:53:06+01:00 at processTicksAndRejections (node:internal/process/task_queues:96:5)
2024-02-22T16:53:06+01:00 at KibanaTransport.request (/usr/share/kibana/src/core/server/elasticsearch/client/create_transport.js:58:16)
2024-02-22T16:53:06+01:00 at Snapshot.get (/usr/share/kibana/node_modules/@elastic/elasticsearch/lib/api/api/snapshot.js:190:16)
2024-02-22T16:53:06+01:00 at /usr/share/kibana/x-pack/plugins/snapshot_restore/server/routes/api/snapshots.js:126:32
2024-02-22T16:53:06+01:00 at Router.handle (/usr/share/kibana/src/core/server/http/router/router.js:163:30)
2024-02-22T16:53:06+01:00 at handler (/usr/share/kibana/src/core/server/http/router/router.js:124:50)
2024-02-22T16:53:06+01:00 at exports.Manager.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/toolkit.js:60:28)
2024-02-22T16:53:06+01:00 at Object.internals.handler (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:46:20)
2024-02-22T16:53:06+01:00 at exports.execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/handler.js:31:20)
2024-02-22T16:53:06+01:00 at Request._lifecycle (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:371:32)
2024-02-22T16:53:06+01:00 at Request._execute (/usr/share/kibana/node_modules/@hapi/hapi/lib/request.js:281:9)

Any idea what causing this?

Thanks in advance

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.