Dear Community,
We are currently at the learning stage of Elasticsearch. We are using Bitbucket Data Centre in a cluster of 5 servers and installed Elasticsearch "7.5.2" in single separated server integrated to Bitbucket using Buckler plugin. For the past several months, we always notice the below error occurring multiple times each day in our Elasticsearch logs and we are unable to find the cause and solution for this error. We already tried running re-index few times but still the errors persist to come again in logs. Please assist us to fix these errors. Also kindly provide some suggestion on how to cleanup an Elasticsearch server to increase free space.
Logs:
[2021-10-05T11:03:09,700][DEBUG][o.e.a.b.TransportShardBulkAction] [xxxxxxxxxx] [bitbucket-search][3] failed to execute bulk item (index) index {[bitbucket-search][_doc][89924__lib/VisuStyle/Hyler_Flat_SP17_1.0.0.0/Images/Hyler_Trademark.svg], source[n/a, actual length: [18.5kb], max length: 2kb]}
java.lang.IndexOutOfBoundsException: Index 1 out of bounds for length 1
at jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:64) ~[?:?]
at jdk.internal.util.Preconditions.outOfBoundsCheckIndex(Preconditions.java:70) ~[?:?]
at jdk.internal.util.Preconditions.checkIndex(Preconditions.java:248) ~[?:?]
at java.util.Objects.checkIndex(Objects.java:373) ~[?:?]
at java.util.ArrayList.get(ArrayList.java:425) ~[?:?]
at org.apache.lucene.analysis.core.FlattenGraphFilter.releaseBufferedToken(FlattenGraphFilter.java:204) ~[lucene-analyzers-common-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:33]
at org.apache.lucene.analysis.core.FlattenGraphFilter.incrementToken(FlattenGraphFilter.java:258) ~[lucene-analyzers-common-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:33]
at org.apache.lucene.index.DefaultIndexingChain$PerField.invert(DefaultIndexingChain.java:812) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]
at org.apache.lucene.index.DefaultIndexingChain.processField(DefaultIndexingChain.java:442) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]
at org.apache.lucene.index.DefaultIndexingChain.processDocument(DefaultIndexingChain.java:406) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]
at org.apache.lucene.index.DocumentsWriterPerThread.updateDocument(DocumentsWriterPerThread.java:250) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]
at org.apache.lucene.index.DocumentsWriter.updateDocument(DocumentsWriter.java:495) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]
at org.apache.lucene.index.IndexWriter.updateDocument(IndexWriter.java:1594) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]
at org.apache.lucene.index.IndexWriter.addDocument(IndexWriter.java:1213) ~[lucene-core-8.3.0.jar:8.3.0 2aa586909b911e66e1d8863aa89f173d69f86cd2 - ishan - 2019-10-25 23:10:03]
at org.elasticsearch.index.engine.InternalEngine.addDocs(InternalEngine.java:1152) ~[elasticsearch-7.5.2.jar:7.5.2]
.
.
.
.
.
.
.
.
.
[2021-10-12T00:10:36,342][DEBUG][o.e.a.s.TransportSearchScrollAction] [fe0vm01228] [3049837] Failed to execute query phase
org.elasticsearch.transport.RemoteTransportException: [xxxxxxxxx][xx.xx.xx.xx:9300][indices:data/read/search[phase/query/scroll]]
Caused by: org.elasticsearch.search.SearchContextMissingException: No search context found for id [3049837]
at org.elasticsearch.search.SearchService.getExecutor(SearchService.java:457) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.search.SearchService.runAsync(SearchService.java:344) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:396) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.SearchTransportService.lambda$registerRequestHandler$9(SearchTransportService.java:349) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.RequestHandlerRegistry.processMessageReceived(RequestHandlerRegistry.java:63) ~[elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.sendLocalRequest(TransportService.java:746) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.access$100(TransportService.java:74) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService$3.sendRequest(TransportService.java:127) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.sendRequestInternal(TransportService.java:692) [elasticsearch-7.5.2.jar:7.5.2]
at com.atlassian.elasticsearch.buckler.AuthorisedSender.sendRequest(AuthorisedSender.java:28) [buckler-plugin-2.1.3.jar:?]
at org.elasticsearch.transport.TransportService.sendRequest(TransportService.java:602) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:646) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.transport.TransportService.sendChildRequest(TransportService.java:637) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.SearchTransportService.sendExecuteScrollQuery(SearchTransportService.java:148) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.SearchScrollQueryThenFetchAsyncAction.executeInitialPhase(SearchScrollQueryThenFetchAsyncAction.java:63) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.SearchScrollAsyncAction.run(SearchScrollAsyncAction.java:198) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.SearchScrollAsyncAction.lambda$run$0(SearchScrollAsyncAction.java:110) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:63) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.SearchScrollAsyncAction.collectNodesAndRun(SearchScrollAsyncAction.java:129) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.SearchScrollAsyncAction.run(SearchScrollAsyncAction.java:110) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.TransportSearchScrollAction.doExecute(TransportSearchScrollAction.java:68) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.search.TransportSearchScrollAction.doExecute(TransportSearchScrollAction.java:35) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.support.TransportAction$RequestFilterChain.proceed(TransportAction.java:153) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:129) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.action.support.TransportAction.execute(TransportAction.java:64) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.client.node.NodeClient.executeLocally(NodeClient.java:83) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.client.node.NodeClient.doExecute(NodeClient.java:72) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.client.support.AbstractClient.execute(AbstractClient.java:396) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.client.support.AbstractClient.searchScroll(AbstractClient.java:544) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.rest.action.search.RestSearchScrollAction.lambda$prepareRequest$1(RestSearchScrollAction.java:72) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.rest.BaseRestHandler.handleRequest(BaseRestHandler.java:108) [elasticsearch-7.5.2.jar:7.5.2]
at com.atlassian.elasticsearch.buckler.RestAuthenticationHandler.handleRequest(RestAuthenticationHandler.java:38) [buckler-plugin-2.1.3.jar:?]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:222) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.rest.RestController.tryAllHandlers(RestController.java:295) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.rest.RestController.dispatchRequest(RestController.java:166) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.http.AbstractHttpServerTransport.dispatchRequest(AbstractHttpServerTransport.java:322) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.http.AbstractHttpServerTransport.handleIncomingRequest(AbstractHttpServerTransport.java:372) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.http.AbstractHttpServerTransport.incomingRequest(AbstractHttpServerTransport.java:301) [elasticsearch-7.5.2.jar:7.5.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:69) [transport-netty4-client-7.5.2.jar:7.5.2]
at org.elasticsearch.http.netty4.Netty4HttpRequestHandler.channelRead0(Netty4HttpRequestHandler.java:31) [transport-netty4-client-7.5.2.jar:7.5.2]
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at org.elasticsearch.http.netty4.Netty4HttpPipeliningHandler.channelRead(Netty4HttpPipeliningHandler.java:58) [transport-netty4-client-7.5.2.jar:7.5.2]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageCodec.channelRead(MessageToMessageCodec.java:111) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:360) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:352) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102) [netty-codec-4.1.43.Final.jar:4.1.43.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:374) [netty-transport-4.1.43.Final.jar:4.1.43.Final]
.
.
.
.
.
.
Thanks,
Lional