Full Collection rebuild for Indexing fails

Boodle · July 2, 2021, 6:20pm

Upgraded to new Oracle12c CMS and migrated ~4 Million documents. However running the Full Collection Rebuild for Indexing fails with below error. 2 difference CMS instances are pointing to same Elasticsearch..

Error:
">(internal)/6 06.22 15:25:40.107 index update work Unable to index metadata only. This may indicate the collection is damaged. Indexing aborted.

systemdatabase/6 06.22 15:25:40.107 index update work !csMonitorActiveDbConnections,0
systemdatabase/6 06.22 15:25:40.107 index update work Connection with id of 'index update work.3' is removed from active connections with key of '482'.
systemdatabase/6 06.22 15:25:40.107 index update work release pool connection
(internal)/3 06.22 15:25:40.107 index update work !csIndexerAbortedMsg intradoc.common.ServiceException: !csIndexerMetaOnlyFailed

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.handleIndexerResults(IndexerBulkLoader.java:601)

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.createBulkLoad(IndexerBulkLoader.java:373)

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.doWork(IndexerBulkLoader.java:170)"

Other traces

taskmanager/6 06.22 06:19:08.173 TextExport_47 task Monitor intradoc.taskmanager.TaskMonitor$1@47756fd8 exiting

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 error reading available bytes from input stream java.io.IOException: Stream closed

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at java.io.BufferedInputStream.available(BufferedInputStream.java:410)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at intradoc.taskmanager.TaskProcessStreamMonitor.readInputStream(TaskProcessStreamMonitor.java:353)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at intradoc.taskmanager.TaskProcessStreamMonitor.runStdErrThread(TaskProcessStreamMonitor.java:456)

Christian_Dahlqvist · July 3, 2021, 2:35pm

What is in the Elasticsearch logs?

Boodle · July 4, 2021, 2:44pm

@Christian_Dahlqvist See below error in ElasticSearch logs:

[{es03}{-Q}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es03][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,546][INFO ][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [es2_internal][1] from [{es02}{}{DNj--_b5C-_XEMNA}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es02][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,548][INFO ][o.e.c.c.Coordinator ] [es01] master node [{es03}{}{}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [es03][disconnected] disconnected
[2021-07-03T19:22:40,558][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) [netty-commo

[2021-07-03T19:22:40,568][WARN ][o.e.t.OutboundHandler ] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}]
org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down.
[2021-07-03T19:22:40,569][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]

[2021-07-03T19:22:40,583][WARN ][o.e.t.OutboundHandler ] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}]
org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down.
at org.elasticsearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:149) [transport-netty4-client-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:133) [elasticsearch-7.10.1.jar:7.10.1]

[2021-07-03T19:22:40,614][INFO ][o.e.n.Node ] [es01] stopped
[2021-07-03T19:22:40,614][INFO ][o.e.n.Node ] [es01] closing ...
[2021-07-03T19:22:40,622][INFO ][o.e.n.Node ] [es01] closed

warkolm · July 6, 2021, 2:59am

That shows something has asked Elasticsearch to shutdown for some reason. Is there more to the log?

Boodle · July 6, 2021, 4:29pm

@warkolm Pls. find complete logs at

github.com

boodlefin/bf/blob/abefa2e0bc04ba262239d3dd942fbbe17b66729c/ElasticSearch_logs.txt

[2021-07-03T19:21:44,091][INFO ][o.e.l.LicenseService     ] [es01] license [] mode [basic] - valid
[2021-07-03T19:21:44,092][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [es01] Active license is now [BASIC]; Security is enabled
[2021-07-03T19:21:44,109][INFO ][o.e.h.AbstractHttpServerTransport] [es01] publish_address {}, bound_addresses {}
[2021-07-03T19:21:44,109][INFO ][o.e.n.Node               ] [es01] started
[2021-07-03T19:22:40,449][INFO ][o.e.n.Node               ] [es01] stopping ...
[2021-07-03T19:22:40,456][INFO ][o.e.x.w.WatcherService   ] [es01] stopping watch service, reason [shutdown initiated]
[2021-07-03T19:22:40,457][INFO ][o.e.x.w.WatcherLifeCycleService] [es01] watcher has stopped and shutdown
[2021-07-03T19:22:40,457][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es01] [controller/208] [Main.cc@154] ML controller exiting
[2021-07-03T19:22:40,457][INFO ][o.e.x.m.p.NativeController] [es01] Native controller process has stopped - no new native processes can be started

[2021-07-03T19:22:40,547][INFO ][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [.monitoring-es-7-2021.06.27][0] from [{es03}{-Q}{}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es03][][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,546][INFO ][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [es2_internal][1] from [{es02}{tNR1wgusSsKoWPjnFhhUmQ}{DNj-2pwkR8-_b5C-_XEMNA}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es02][][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,548][INFO ][o.e.c.c.Coordinator      ] [es01] master node [{es03}{-Q}{}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [es03][][disconnected] disconnected
[2021-07-03T19:22:40,558][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]

This file has been truncated. show original

Boodle · July 8, 2021, 4:20pm

@Christian_Dahlqvist @warkolm please find logs as requested. Can you help please @DavidTurner. I saw these discussions D1 , D2 . Is it something similar? Can you please share the resolution. here is docker-compose_yml

system · August 5, 2021, 4:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.