ECK elasticsearch - 500

Elastic Orchestration Elastic Cloud on Kubernetes (ECK)
1

I had elasticsearch running in a cluster and all was well... however now I get 500s and the logs are full of stack traces. The first few lines:

 "at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:65) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:545) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.ClientScrollableHitSource.doStart(ClientScrollableHitSource.java:77) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.ScrollableHitSource.start(ScrollableHitSource.java:78) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.AbstractAsyncBulkByScrollAction.start(AbstractAsyncBulkByScrollAction.java:242) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.lambda$doExecute$0(TransportUpdateByQueryAction.java:75) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.executeSlicedAction(BulkByScrollParallelizationHelper.java:105) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper$1.onResponse(BulkByScrollParallelizationHelper.java:74) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper$1.onResponse(BulkByScrollParallelizationHelper.java:71) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.initTaskState(BulkByScrollParallelizationHelper.java:142) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.startSlicedAction(BulkByScrollParallelizationHelper.java:71) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.doExecute(TransportUpdateByQueryAction.java:68) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.doExecute(TransportUpdateByQueryAction.java:47) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:87) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:173) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:365) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) [x-pack-security-7
    .10.1.jar:7.10.1]",
2

Can you share some more of the exceptions you are seeing? This snipped you shared contains no clue as to what is going on (at least for me) but I have the feeling there are more clues in the logs of the Elasticsearch nodes.

It would also be good to know what the ECK operator thinks about the health of your cluster. So the output of kubectl get es <your-elasticsearch-cluster-name> would be interesting.

3

I think the problem was that at least one of the pvcs was full - I found something in the logs about no disk space. I've reconfigured with a new nodeset and larger volumeClaimTemplate, and so far all is well. I guess I need to have something in place to monitor for this.

At the time I recall that the operator was reporting that the health was unknown, but I'm not 100% sure.