# Full Collection rebuild for Indexing fails **URL:** \ **Category:** Elasticsearch\ **Tags:** docker\ **Created:** [July 2, 2021, 6:20pm UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679 "2021-07-02T18:20:10Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Boodle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boodle/32/86046_2.png) [@Boodle](https://discuss.elastic.co/u/Boodle)\ **Post date:** [July 2, 2021, 6:20pm UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679/1 "2021-07-02T18:20:11Z")
Upgraded to new Oracle12c CMS and migrated ~4 Million documents. However running the Full Collection Rebuild for Indexing fails with below error. 2 difference CMS instances are pointing to same Elasticsearch.. **Error:** "\>(internal)/6 06.22 15:25:40.107 index update work Unable to index metadata only. This may indicate the collection is damaged. Indexing aborted. > systemdatabase/6 06.22 15:25:40.107 index update work !csMonitorActiveDbConnections,0 > systemdatabase/6 06.22 15:25:40.107 index update work Connection with id of 'index update work.3' is removed from active connections with key of '482'. > systemdatabase/6 06.22 15:25:40.107 index update work release pool connection > (internal)/3 06.22 15:25:40.107 index update work !csIndexerAbortedMsg intradoc.common.ServiceException: !csIndexerMetaOnlyFailed (internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.handleIndexerResults(IndexerBulkLoader.java:601) (internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.createBulkLoad(IndexerBulkLoader.java:373) (internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.doWork(IndexerBulkLoader.java:170)" **Other traces** > taskmanager/6 06.22 06:19:08.173 TextExport\_47 task Monitor [intradoc.taskmanager.TaskMonitor$1@47756fd8](mailto:intradoc.taskmanager.TaskMonitor$1@47756fd8) exiting > taskmanager/7 06.22 06:19:08.173 TaskLauncher\_TextExport\_stderr\_\_47 error reading available bytes from input stream java.io.IOException: Stream closed taskmanager/7 06.22 06:19:08.173 TaskLauncher\_TextExport\_stderr\_\_47 at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159) taskmanager/7 06.22 06:19:08.173 TaskLauncher\_TextExport\_stderr\_\_47 at java.io.BufferedInputStream.available(BufferedInputStream.java:410) taskmanager/7 06.22 06:19:08.173 TaskLauncher\_TextExport\_stderr\_\_47 at java.io.BufferedInputStream.available(BufferedInputStream.java:410) taskmanager/7 06.22 06:19:08.173 TaskLauncher\_TextExport\_stderr\_\_47 at intradoc.taskmanager.TaskProcessStreamMonitor.readInputStream(TaskProcessStreamMonitor.java:353) taskmanager/7 06.22 06:19:08.173 TaskLauncher\_TextExport\_stderr\_\_47 at intradoc.taskmanager.TaskProcessStreamMonitor.runStdErrThread(TaskProcessStreamMonitor.java:456) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 3, 2021, 2:35pm UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679/2 "2021-07-03T14:35:45Z")
What is in the Elasticsearch logs? ---
**Author:** ![Boodle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boodle/32/86046_2.png) [@Boodle](https://discuss.elastic.co/u/Boodle)\ **Post date:** [July 4, 2021, 2:44pm UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679/3 "2021-07-04T14:44:39Z")
@Christian_Dahlqvist See below error in ElasticSearch logs: [{es03}{-Q}{}{}{cdhilmrstw}{ml.machine\_memory=25010077696, ml.max\_open\_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es03][internal:index/shard/recovery/start\_recovery] disconnected] [2021-07-03T19:22:40,546][INFO][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [es2\_internal][1] from [{es02}{}{DNj--\_b5C-\_XEMNA}{}{cdhilmrstw}{ml.machine\_memory=25010077696, ml.max\_open\_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es02][internal:index/shard/recovery/start\_recovery] disconnected] [2021-07-03T19:22:40,548][INFO][o.e.c.c.Coordinator] [es01] master node [{es03}{}{}{}{}{cdhilmrstw}{ml.machine\_memory=25010077696, ml.max\_open\_jobs=20, xpack.installed=true, transform.node=true}] failed, restarting discovery org.elasticsearch.transport.NodeDisconnectedException: [es03][disconnected] disconnected [2021-07-03T19:22:40,558][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) [netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) [netty-commo [2021-07-03T19:22:40,568][WARN][o.e.t.OutboundHandler] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}] org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down. [2021-07-03T19:22:40,569][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down? java.util.concurrent.RejectedExecutionException: event executor terminated at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final] at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final] [2021-07-03T19:22:40,583][WARN][o.e.t.OutboundHandler] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}] org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down. at org.elasticsearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:149) [transport-netty4-client-7.10.1.jar:7.10.1] at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:133) [elasticsearch-7.10.1.jar:7.10.1] [2021-07-03T19:22:40,614][INFO][o.e.n.Node] [es01] stopped [2021-07-03T19:22:40,614][INFO][o.e.n.Node] [es01] closing ... [2021-07-03T19:22:40,622][INFO][o.e.n.Node] [es01] closed ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [July 6, 2021, 2:59am UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679/4 "2021-07-06T02:59:25Z")
That shows something has asked Elasticsearch to shutdown for some reason. Is there more to the log? ---
**Author:** ![Boodle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boodle/32/86046_2.png) [@Boodle](https://discuss.elastic.co/u/Boodle)\ **Post date:** [July 6, 2021, 4:29pm UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679/5 "2021-07-06T16:29:18Z")
@warkolm Pls. find complete logs at > ---
**Author:** ![Boodle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/boodle/32/86046_2.png) [@Boodle](https://discuss.elastic.co/u/Boodle)\ **Post date:** [July 8, 2021, 4:20pm UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679/6 "2021-07-08T16:20:07Z")
@Christian_Dahlqvist @warkolm please find logs as requested. Can you help please @DavidTurner. I saw these discussions [D1](https://discuss.elastic.co/t/master-role-does-not-switch-automaticaly/214213/16) , D2 . Is it something similar? Can you please share the resolution. here is [docker-compose\_yml](https://github.com/boodlefin/bf/blob/main/docker-compose_yml_file.txt) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 5, 2021, 4:20pm UTC](https://discuss.elastic.co/t/full-collection-rebuild-for-indexing-fails/277679/7 "2021-08-05T16:20:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.