ECK elasticsearch - 500

PaulRudin · January 18, 2021, 6:56pm

I had elasticsearch running in a cluster and all was well... however now I get 500s and the logs are full of stack traces. The first few lines:

 "at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:65) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:545) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.ClientScrollableHitSource.doStart(ClientScrollableHitSource.java:77) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.ScrollableHitSource.start(ScrollableHitSource.java:78) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.AbstractAsyncBulkByScrollAction.start(AbstractAsyncBulkByScrollAction.java:242) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.lambda$doExecute$0(TransportUpdateByQueryAction.java:75) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.executeSlicedAction(BulkByScrollParallelizationHelper.java:105) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper$1.onResponse(BulkByScrollParallelizationHelper.java:74) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper$1.onResponse(BulkByScrollParallelizationHelper.java:71) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.initTaskState(BulkByScrollParallelizationHelper.java:142) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.startSlicedAction(BulkByScrollParallelizationHelper.java:71) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.doExecute(TransportUpdateByQueryAction.java:68) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.doExecute(TransportUpdateByQueryAction.java:47) [reindex-client-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:87) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:173) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:365) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) [x-pack-security-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.10.1.jar:7.10.1]",
    "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) [x-pack-security-7
    .10.1.jar:7.10.1]",

pebrc · January 19, 2021, 2:00pm

Can you share some more of the exceptions you are seeing? This snipped you shared contains no clue as to what is going on (at least for me) but I have the feeling there are more clues in the logs of the Elasticsearch nodes.

It would also be good to know what the ECK operator thinks about the health of your cluster. So the output of kubectl get es <your-elasticsearch-cluster-name> would be interesting.

PaulRudin · January 19, 2021, 3:46pm

I think the problem was that at least one of the pvcs was full - I found something in the logs about no disk space. I've reconfigured with a new nodeset and larger volumeClaimTemplate, and so far all is well. I guess I need to have something in place to monitor for this.

At the time I recall that the operator was reporting that the health was unknown, but I'm not 100% sure.

Topic		Replies	Views
Elasticsearch cluster on ECK issues (503 Service Unavailable) Elastic Cloud on Kubernetes (ECK)	0	51	July 17, 2024
Elastic Cloud on Kubernates Elastic Cloud on Kubernetes (ECK) elastic-stack-monitoring	2	319	June 18, 2023
ECK continuous log spamming Elastic Cloud on Kubernetes (ECK)	2	527	June 28, 2023
How to secure elasticsearch on kubernetes Elasticsearch elastic-stack-security	4	974	July 20, 2022
After creating Elasticsearch, and then changing the yaml file, apply fails Elastic Cloud on Kubernetes (ECK)	3	1115	March 25, 2021

ECK elasticsearch - 500

Related topics