We're running an ES cluster using cloud-on-k8s and when I try and create a snapshot using the Kibana UI I get the following errors from elasticsearch and kibana:
portal-elastic-es-main-0 elasticsearch {"type": "server", "timestamp": "2020-03-05T12:23:14,224Z", "level": "WARN", "component": "r.suppressed", "cluster.name": "portal-elastic", "node.name": "portal-elastic-es-main-0", "message": "path: /_slm/policy/daily/_execute, params: {name=daily}", "cluster.uuid": "AY3K33D1TZ6NeQ_tsYLfmA", "node.id": "DKHkJG7HQBS8KisfmfxAdg" ,
portal-elastic-es-main-0 elasticsearch "stacktrace": ["org.elasticsearch.transport.RemoteTransportException: [portal-elastic-es-main-2][10.52.4.33:9300][cluster:admin/slm/execute]",
portal-elastic-es-main-0 elasticsearch "Caused by: java.lang.NullPointerException",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.xpack.core.slm.SnapshotLifecyclePolicy.toRequest(SnapshotLifecyclePolicy.java:258) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.xpack.slm.SnapshotLifecycleTask.lambda$maybeTakeSnapshot$1(SnapshotLifecycleTask.java:86) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at java.util.Optional.map(Optional.java:258) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.xpack.slm.SnapshotLifecycleTask.maybeTakeSnapshot(SnapshotLifecycleTask.java:85) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.xpack.slm.action.TransportExecuteSnapshotLifecycleAction.masterOperation(TransportExecuteSnapshotLifecycleAction.java:80) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.xpack.slm.action.TransportExecuteSnapshotLifecycleAction.masterOperation(TransportExecuteSnapshotLifecycleAction.java:35) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.action.support.master.TransportMasterNodeAction.masterOperation(TransportMasterNodeAction.java:98) ~[elasticsearch-7.5.2.jar:7.5.2]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.action.support.master.TransportMasterNodeAction$AsyncSingleAction.lambda$doStart$3(TransportMasterNodeAction.java:169) ~[elasticsearch-7.5.2.jar:7.5.2]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.action.ActionRunnable$2.doRun(ActionRunnable.java:73) ~[elasticsearch-7.5.2.jar:7.5.2]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:773) ~[elasticsearch-7.5.2.jar:7.5.2]",
portal-elastic-es-main-0 elasticsearch "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.5.2.jar:7.5.2]",
portal-elastic-es-main-0 elasticsearch "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]",
portal-elastic-es-main-0 elasticsearch "at java.lang.Thread.run(Thread.java:830) [?:?]"] }
portal-elastic-kb-75cb599c99-72f5h kibana {"type":"error","@timestamp":"2020-03-05T12:23:14Z","tags":[],"pid":6,"level":"error","error":{"message":"[remote_transport_exception] [portal-elastic-es-main-2][10.52.4.33:9300][cluster:admin/slm/execute]","name":"Error","stack":"Error: [remote_transport_exception] [portal-elastic-es-main-2][10.52.4.33:9300][cluster:admin/slm/execute]\n at respond (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:349:15)\n at checkRespForFailure (/usr/share/kibana/node_modules/elasticsearch/src/lib/transport.js:306:7)\n at HttpConnector.<anonymous> (/usr/share/kibana/node_modules/elasticsearch/src/lib/connectors/http.js:173:7)\n at IncomingMessage.wrapper (/usr/share/kibana/node_modules/elasticsearch/node_modules/lodash/lodash.js:4929:19)\n at IncomingMessage.emit (events.js:194:15)\n at endReadableNT (_stream_readable.js:1103:12)\n at process._tickCallback (internal/process/next_tick.js:63:19)"},"url":{"protocol":null,"slashes":null,"auth":null,"host":null,"port":null,"hostname":null,"hash":null,"search":null,"query":{},"pathname":"/api/snapshot_restore/policy/daily/run","path":"/api/snapshot_restore/policy/daily/run","href":"/api/snapshot_restore/policy/daily/run"},"message":"[remote_transport_exception] [portal-elastic-es-main-2][10.52.4.33:9300][cluster:admin/slm/execute]"}
I have shelled into the portal-elastic-es-main-0 pod and created a snapshot with curl which ran with no problems:
curl -k -X PUT https://10.52.4.33:9200/_snapshot/portal-backups-bucket/snapshot_1
I realise this is a different port but wanted to make sure it wasn't the actual snapshot creation itself causing issues.
Any ideas?