Full Collection rebuild for Indexing fails

Boodle · July 2, 2021, 6:20pm

Upgraded to new Oracle12c CMS and migrated ~4 Million documents. However running the Full Collection Rebuild for Indexing fails with below error. 2 difference CMS instances are pointing to same Elasticsearch..

Error:
">(internal)/6 06.22 15:25:40.107 index update work Unable to index metadata only. This may indicate the collection is damaged. Indexing aborted.

systemdatabase/6 06.22 15:25:40.107 index update work !csMonitorActiveDbConnections,0
systemdatabase/6 06.22 15:25:40.107 index update work Connection with id of 'index update work.3' is removed from active connections with key of '482'.
systemdatabase/6 06.22 15:25:40.107 index update work release pool connection
(internal)/3 06.22 15:25:40.107 index update work !csIndexerAbortedMsg intradoc.common.ServiceException: !csIndexerMetaOnlyFailed

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.handleIndexerResults(IndexerBulkLoader.java:601)

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.createBulkLoad(IndexerBulkLoader.java:373)

(internal)/3 06.22 15:25:40.107 index update work at intradoc.indexer.IndexerBulkLoader.doWork(IndexerBulkLoader.java:170)"

Other traces

taskmanager/6 06.22 06:19:08.173 TextExport_47 task Monitor intradoc.taskmanager.TaskMonitor$1@47756fd8 exiting

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 error reading available bytes from input stream java.io.IOException: Stream closed

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at java.io.BufferedInputStream.getInIfOpen(BufferedInputStream.java:159)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at java.io.BufferedInputStream.available(BufferedInputStream.java:410)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at intradoc.taskmanager.TaskProcessStreamMonitor.readInputStream(TaskProcessStreamMonitor.java:353)

taskmanager/7 06.22 06:19:08.173 TaskLauncher_TextExport_stderr__47 at intradoc.taskmanager.TaskProcessStreamMonitor.runStdErrThread(TaskProcessStreamMonitor.java:456)

Christian_Dahlqvist · July 3, 2021, 2:35pm

What is in the Elasticsearch logs?

Boodle · July 4, 2021, 2:44pm

@Christian_Dahlqvist See below error in ElasticSearch logs:

[{es03}{-Q}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es03][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,546][INFO ][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [es2_internal][1] from [{es02}{}{DNj--_b5C-_XEMNA}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es02][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,548][INFO ][o.e.c.c.Coordinator ] [es01] master node [{es03}{}{}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [es03][disconnected] disconnected
[2021-07-03T19:22:40,558][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:818) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.safeExecute(DefaultPromise.java:841) [netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.DefaultPromise.notifyListeners(DefaultPromise.java:498) [netty-commo

[2021-07-03T19:22:40,568][WARN ][o.e.t.OutboundHandler ] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}]
org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down.
[2021-07-03T19:22:40,569][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]

[2021-07-03T19:22:40,583][WARN ][o.e.t.OutboundHandler ] [es01] send message failed [channel: Netty4TcpChannel{localAddress=/, remoteAddress=/}]
org.elasticsearch.transport.TransportException: Cannot send message, event loop is shutting down.
at org.elasticsearch.transport.netty4.Netty4TcpChannel.sendMessage(Netty4TcpChannel.java:149) [transport-netty4-client-7.10.1.jar:7.10.1]
at org.elasticsearch.transport.OutboundHandler.internalSend(OutboundHandler.java:133) [elasticsearch-7.10.1.jar:7.10.1]

[2021-07-03T19:22:40,614][INFO ][o.e.n.Node ] [es01] stopped
[2021-07-03T19:22:40,614][INFO ][o.e.n.Node ] [es01] closing ...
[2021-07-03T19:22:40,622][INFO ][o.e.n.Node ] [es01] closed

warkolm · July 6, 2021, 2:59am

That shows something has asked Elasticsearch to shutdown for some reason. Is there more to the log?

Boodle · July 6, 2021, 4:29pm

@warkolm Pls. find complete logs at

github.com

boodlefin/bf/blob/abefa2e0bc04ba262239d3dd942fbbe17b66729c/ElasticSearch_logs.txt

[2021-07-03T19:21:44,091][INFO ][o.e.l.LicenseService     ] [es01] license [] mode [basic] - valid
[2021-07-03T19:21:44,092][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [es01] Active license is now [BASIC]; Security is enabled
[2021-07-03T19:21:44,109][INFO ][o.e.h.AbstractHttpServerTransport] [es01] publish_address {}, bound_addresses {}
[2021-07-03T19:21:44,109][INFO ][o.e.n.Node               ] [es01] started
[2021-07-03T19:22:40,449][INFO ][o.e.n.Node               ] [es01] stopping ...
[2021-07-03T19:22:40,456][INFO ][o.e.x.w.WatcherService   ] [es01] stopping watch service, reason [shutdown initiated]
[2021-07-03T19:22:40,457][INFO ][o.e.x.w.WatcherLifeCycleService] [es01] watcher has stopped and shutdown
[2021-07-03T19:22:40,457][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [es01] [controller/208] [Main.cc@154] ML controller exiting
[2021-07-03T19:22:40,457][INFO ][o.e.x.m.p.NativeController] [es01] Native controller process has stopped - no new native processes can be started

[2021-07-03T19:22:40,547][INFO ][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [.monitoring-es-7-2021.06.27][0] from [{es03}{-Q}{}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es03][][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,546][INFO ][o.e.i.r.PeerRecoveryTargetService] [es01] recovery of [es2_internal][1] from [{es02}{tNR1wgusSsKoWPjnFhhUmQ}{DNj-2pwkR8-_b5C-_XEMNA}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] interrupted by network disconnect, will retry in [5s]; cause: [[es02][][internal:index/shard/recovery/start_recovery] disconnected]
[2021-07-03T19:22:40,548][INFO ][o.e.c.c.Coordinator      ] [es01] master node [{es03}{-Q}{}{}{}{cdhilmrstw}{ml.machine_memory=25010077696, ml.max_open_jobs=20, xpack.installed=true, transform.node=true}] failed, restarting discovery
org.elasticsearch.transport.NodeDisconnectedException: [es03][][disconnected] disconnected
[2021-07-03T19:22:40,558][ERROR][i.n.u.c.D.rejectedExecution] [es01] Failed to submit a listener notification task. Event loop shut down?
java.util.concurrent.RejectedExecutionException: event executor terminated
        at io.netty.util.concurrent.SingleThreadEventExecutor.reject(SingleThreadEventExecutor.java:926) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.offerTask(SingleThreadEventExecutor.java:353) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.addTask(SingleThreadEventExecutor.java:346) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]
        at io.netty.util.concurrent.SingleThreadEventExecutor.execute(SingleThreadEventExecutor.java:828) ~[netty-common-4.1.49.Final.jar:4.1.49.Final]

This file has been truncated. show original

Boodle · July 8, 2021, 4:20pm

@Christian_Dahlqvist @warkolm please find logs as requested. Can you help please @DavidTurner. I saw these discussions D1 , D2 . Is it something similar? Can you please share the resolution. here is docker-compose_yml

system · August 5, 2021, 4:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
During the backup some indices fails giving me this error Elasticsearch snapshot-and-restore	3	333	March 11, 2023
Elasticsearch running on Docker failing in taking snapshots and restore Elasticsearch docker	7	1071	April 28, 2020
Trying to upgrade elasticsearch docker from 1.5.2 to 2.4.6 Elasticsearch docker	6	396	March 10, 2022
IndexFormatException: ES? Elasticsearch	3	557	July 6, 2017
Multiple index templates may not match during index creation Elasticsearch docker	2	208	July 18, 2023

Full Collection rebuild for Indexing fails

Related topics