Full Collection rebuild for Indexing fails

Upgraded to new Oracle12c CMS and migrated ~4 Million documents. However running the Full Collection Rebuild for Indexing fails with below error. 2 difference CMS instances are pointing to same ElasticSearch..

Error:
">(internal)/6 06.22 15:25:40.107 index update work Unable to index metadata only. This may indicate the collection is damaged. Indexing aborted.

systemdatabase/6 06.22 15:25:40.107 index update work !csMonitorActiveDbConnections,0
systemdatabase/6 06.22 15:25:40.107 index update work Connection with id of 'index update work.3' is removed from active connections with key of '482'.
systemdatabase/6 06.22 15:25:40.107 index update work release pool connection
(internal)/3 06.22 15:25:40.107 index update work !csIndexerAbortedMsg intradoc.common.ServiceException: !csIndexerMetaOnlyFailed

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.handleIndexerResults(IndexerBulkLoader.java:601)

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.createBulkLoad(IndexerBulkLoader.java:373)

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.doWork(IndexerBulkLoader.java:170)"

Other traces

taskmanager/6 06.22 06:19:08.173 TextExport_47 task Monitor intradoc.taskmanager.TaskMonitor$1@47756fd8 exiting

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 error reading available bytes from input stream java.io.IOException: Stream closed

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at java.io.BufferedInputStream.available(BufferedInputStream.java:410)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at java.io.BufferedInputStream.available(BufferedInputStream.java:410)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at intradoc.taskmanager.TaskProcessStreamMonitor.readInputStream(TaskProcessStreamMonitor.java:353)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at intradoc.taskmanager.TaskProcessStreamMonitor.runStdErrThread(TaskProcessStreamMonitor.java:456)

What is in the Elasticsearch logs?

@Christian_Dahlqvist See below error in ElasticSearch logs:

[{es03}{-Q}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es03][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,546][INFO ][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [es2_internal][1] from [{es02}{}{DNj--_b5C-_XEMNA}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es02][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,548][INFO ][o.e.c.c.Coordinator ] [es01] master node [{es03}{}{}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [es03][disconnected] disconnected
[2021-07-03T19:22:40,558][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) [netty-commo

[2021-07-03T19:22:40,568][WARN ][o.e.t.OutboundHandler ] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}]
org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down.
[2021-07-03T19:22:40,569][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]

[2021-07-03T19:22:40,583][WARN ][o.e.t.OutboundHandler ] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}]
org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down.
at org.elasticsearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:149) [transport-netty4-client-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:133) [elasticsearch-7.10.1.jar:7.10.1]

[2021-07-03T19:22:40,614][INFO ][o.e.n.Node ] [es01] stopped
[2021-07-03T19:22:40,614][INFO ][o.e.n.Node ] [es01] closing ...
[2021-07-03T19:22:40,622][INFO ][o.e.n.Node ] [es01] closed

That shows something has asked Elasticsearch to shutdown for some reason. Is there more to the log?

@warkolm Pls. find complete logs at

@Christian_Dahlqvist @warkolm please find logs as requested. Can you help please @DavidTurner. I saw these discussions D1 , D2 . Is it something similar? Can you please share the resolution. here is docker-compose_yml

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.