Hello, we're having a recent issue with reindexing that is failing after a short period of time. We recently discovered a mistake with our original index that only had 1 shard set upon creation when we should've had 20. Given that, we wanted to reindex to address this issue. Both indices have the same exact mappings with the only difference being the shard sizes.
We are currently running a self-hosted Elasticsearch instance on version 8.7.1.
Reindexing command:
POST _reindex/?slices=auto&wait_for_completion=false
{
"source": {
"index": "resource-qualified-media-flattened",
"size": 1000
},
"dest": {
"index": "resource-qualified-media-flattened-generic",
"op_type": "create"
},
"conflicts": "proceed"
}
I've tried various source sizes (5000, 2000, 500, 100, etc) and added/removed the slices param but to no avail. (There is only 1 shard though so I don't think that parameter will matter much in this case) Setting "conflicts": "proceed" seems to not be able to mitigate this issue. I will try the throttling param shortly to see if that makes any difference.
It runs fine until after a short period of time and then throws the following:
{
"completed": true,
"task": {
"node": "M7kOXAuOTZausmS5bedjng",
"id": 39807729,
"type": "transport",
"action": "indices:data/write/reindex",
"status": {
"total": 359061220,
"updated": 0,
"created": 10959000,
"deleted": 0,
"batches": 21918,
"version_conflicts": 0,
"noops": 0,
"retries": {
"bulk": 0,
"search": 0
},
"throttled_millis": 0,
"requests_per_second": -1,
"throttled_until_millis": 0
},
"description": "reindex from [resource-qualified-media-flattened] to [resource-qualified-media-flattened-generic]",
"start_time_in_millis": 1696365346781,
"running_time_in_nanos": 1361131871006,
"cancellable": true,
"cancelled": false,
"headers": {
"trace.id": "2202163dd1df44463cc5e34dddecaa67"
}
},
"error": {
"type": "search_phase_execution_exception",
"reason": "all shards failed",
"phase": "query",
"grouped": true,
"failed_shards": [
{
"shard": -1,
"index": null,
"reason": {
"type": "array_index_out_of_bounds_exception",
"reason": "arraycopy: length -9 is negative"
}
}
],
"caused_by": {
"type": "array_index_out_of_bounds_exception",
"reason": "arraycopy: length -9 is negative"
}
}
}
When checking the Elasticsearch logs:
[2023-10-04T13:49:49,142][WARN ][o.e.t.LoggingTaskListener] [es-client-01] 37883929 failed with exception
org.elasticsearch.action.search.SearchPhaseExecutionException: all shards failed
at org.elasticsearch.action.search.SearchScrollAsyncAction.onShardFailure(SearchScrollAsyncAction.java:290) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.SearchScrollAsyncAction$1.onFailure(SearchScrollAsyncAction.java:190) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListenerResponseHandler.handleException(ActionListenerResponseHandler.java:48) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.search.SearchTransportService$ConnectionCountingHandler.handleException(SearchTransportService.java:644) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.TransportService$UnregisterChildTransportResponseHandler.handleException(TransportService.java:1651) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.TransportService$ContextRestoreResponseHandler.handleException(TransportService.java:1377) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundHandler.doHandleException(InboundHandler.java:442) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundHandler.handleException(InboundHandler.java:429) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundHandler.handlerResponseError(InboundHandler.java:420) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundHandler.messageReceived(InboundHandler.java:142) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundHandler.inboundMessage(InboundHandler.java:97) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.TcpTransport.inboundMessage(TcpTransport.java:829) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundPipeline.forwardFragments(InboundPipeline.java:150) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundPipeline.doHandleBytes(InboundPipeline.java:121) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.InboundPipeline.handleBytes(InboundPipeline.java:86) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.netty4.Netty4MessageInboundHandler.channelRead(Netty4MessageInboundHandler.java:63) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.logging.LoggingHandler.channelRead(LoggingHandler.java:280) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:442) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:103) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1373) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1236) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1285) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Caused by: org.elasticsearch.transport.RemoteTransportException: [es-data-def-0][123.123.123.123:9300][indices:data/read/search[phase/query+fetch/scroll]]
Caused by: java.lang.ArrayIndexOutOfBoundsException: arraycopy: length -9 is negative
at org.elasticsearch.common.io.stream.RecyclerBytesStreamOutput.writeBytes(RecyclerBytesStreamOutput.java:86) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.io.stream.StreamOutput.writeInt(StreamOutput.java:202) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.io.stream.RecyclerBytesStreamOutput.writeInt(RecyclerBytesStreamOutput.java:105) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.io.stream.StreamOutput.writeFloat(StreamOutput.java:449) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.search.SearchHit.writeTo(SearchHit.java:235) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.io.stream.StreamOutput.lambda$writeArray$31(StreamOutput.java:956) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.io.stream.StreamOutput.writeArray(StreamOutput.java:933) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.io.stream.StreamOutput.writeArray(StreamOutput.java:956) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.search.SearchHits.writeTo(SearchHits.java:100) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.search.fetch.FetchSearchResult.writeTo(FetchSearchResult.java:53) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.search.fetch.QueryFetchSearchResult.writeTo(QueryFetchSearchResult.java:73) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.search.fetch.ScrollQueryFetchSearchResult.writeTo(ScrollQueryFetchSearchResult.java:64) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.OutboundMessage.serialize(OutboundMessage.java:70) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.OutboundHandler.sendMessage(OutboundHandler.java:178) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.OutboundHandler.sendResponse(OutboundHandler.java:138) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.TcpTransportChannel.sendResponse(TcpTransportChannel.java:57) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.transport.TaskTransportChannel.sendResponse(TaskTransportChannel.java:41) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.ChannelActionListener.lambda$onResponse$0(ChannelActionListener.java:38) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionListener.run(ActionListener.java:567) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:38) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.support.ChannelActionListener.onResponse(ChannelActionListener.java:20) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:1131) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:50) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionRunnable$2.accept(ActionRunnable.java:47) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.action.ActionRunnable$3.doRun(ActionRunnable.java:72) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:958) ~[elasticsearch-8.7.1.jar:?]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.7.1.jar:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?]
at java.lang.Thread.run(Thread.java:1623) ~[?:?]
Any thoughts?
I did find one other topic with a similar issue but no response was given on it: Remote Reindex from a datastream to another index