# ECK elasticsearch - 500 **URL:** \ **Category:** Elastic Cloud on Kubernetes (ECK)\ **Created:** [January 18, 2021, 6:56pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-500/261462 "2021-01-18T18:56:02Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![PaulRudin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paulrudin/32/82177_2.png) [@PaulRudin](https://discuss.elastic.co/u/PaulRudin)\ **Post date:** [January 18, 2021, 6:56pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-500/261462/1 "2021-01-18T18:56:02Z")
I had elasticsearch running in a cluster and all was well... however now I get 500s and the logs are full of stack traces. The first few lines: ```auto "at org.elasticsearch.xpack.security.authc.AuthenticationService.authenticate(AuthenticationService.java:156) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.applyInternal(SecurityActionFilter.java:156) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.apply(SecurityActionFilter.java:108) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:155) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:83) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:86) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:75) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.client.FilterClient.doExecute(FilterClient.java:65) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.client.ParentTaskAssigningClient.doExecute(ParentTaskAssigningClient.java:63) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:412) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.client.support.AbstractClient.search(AbstractClient.java:545) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.ClientScrollableHitSource.doStart(ClientScrollableHitSource.java:77) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.ScrollableHitSource.start(ScrollableHitSource.java:78) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.AbstractAsyncBulkByScrollAction.start(AbstractAsyncBulkByScrollAction.java:242) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.lambda$doExecute$0(TransportUpdateByQueryAction.java:75) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.executeSlicedAction(BulkByScrollParallelizationHelper.java:105) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper$1.onResponse(BulkByScrollParallelizationHelper.java:74) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper$1.onResponse(BulkByScrollParallelizationHelper.java:71) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.initTaskState(BulkByScrollParallelizationHelper.java:142) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.BulkByScrollParallelizationHelper.startSlicedAction(BulkByScrollParallelizationHelper.java:71) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.doExecute(TransportUpdateByQueryAction.java:68) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.index.reindex.TransportUpdateByQueryAction.doExecute(TransportUpdateByQueryAction.java:47) [reindex-client-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:179) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.support.ActionFilter$Simple.apply(ActionFilter.java:53) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:177) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$apply$0(SecurityActionFilter.java:87) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.action.filter.SecurityActionFilter.lambda$authorizeRequest$4(SecurityActionFilter.java:173) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$15(AuthorizationService.java:365) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]", "at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.authz.interceptor.ResizeRequestInterceptor.intercept(ResizeRequestInterceptor.java:86) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.authz.AuthorizationService.lambda$runRequestInterceptors$14(AuthorizationService.java:360) [x-pack-security-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture$1.doRun(ListenableFuture.java:112) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.EsExecutors$DirectExecutorService.execute(EsExecutors.java:224) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.notifyListener(ListenableFuture.java:106) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.lambda$done$0(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]", "at java.util.ArrayList.forEach(ArrayList.java:1511) [?:?]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.done(ListenableFuture.java:98) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.BaseFuture.set(BaseFuture.java:144) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.common.util.concurrent.ListenableFuture.onResponse(ListenableFuture.java:127) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.StepListener.innerOnResponse(StepListener.java:62) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.action.NotifyOnceListener.onResponse(NotifyOnceListener.java:40) [elasticsearch-7.10.1.jar:7.10.1]", "at org.elasticsearch.xpack.security.authz.interceptor.FieldAndDocumentLevelSecurityRequestInterceptor.intercept(FieldAndDocumentLevelSecurityRequestInterceptor.java:65) [x-pack-security-7 .10.1.jar:7.10.1]", ``` ---
**Author:** ![pebrc](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pebrc/32/101790_2.png) [@pebrc](https://discuss.elastic.co/u/pebrc)\ **Post date:** [January 19, 2021, 2:00pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-500/261462/2 "2021-01-19T14:00:19Z")
Can you share some more of the exceptions you are seeing? This snipped you shared contains no clue as to what is going on (at least for me) but I have the feeling there are more clues in the logs of the Elasticsearch nodes. It would also be good to know what the ECK operator thinks about the health of your cluster. So the output of `kubectl get es ` would be interesting. > **[Troubleshooting methods | Elastic Cloud on Kubernetes \[1.3\] | Elastic](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-troubleshooting-methods.html)** ---
**Author:** ![PaulRudin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paulrudin/32/82177_2.png) [@PaulRudin](https://discuss.elastic.co/u/PaulRudin)\ **Post date:** [January 19, 2021, 3:46pm UTC](https://discuss.elastic.co/t/eck-elasticsearch-500/261462/3 "2021-01-19T15:46:02Z")
I think the problem was that at least one of the pvcs was full - I found something in the logs about no disk space. I've reconfigured with a new nodeset and larger volumeClaimTemplate, and so far all is well. I guess I need to have something in place to monitor for this. At the time I recall that the operator was reporting that the health was unknown, but I'm not 100% sure. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 4, 2022, 8:21am UTC](https://discuss.elastic.co/t/eck-elasticsearch-500/261462/4 "2022-11-04T08:21:55Z")