Hello,
I am trying to upgrade elasticsearch and kibana to 7.16.3 from version 7.9.2.
Below are the steps executed.
- Created a new elasticsearch cluster with 7.16.3 version of elasticsearch and 7.16.3 version of Kibana.
- Restored .kibana_1 and .kibana_task_manager_1 from older elatsicsearch cluster (7.9.2) to new cluster
- Since the dashboards and index patterns was not created after restoring these indices I have decided to manually export the saved_objects from older version and import it in new cluster.
- After importing dashboards are created properly as expected.
- I have restored the data to the new cluster and everything looks good.
- After two days, Kibana service is failed with below message
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: <--- Last few GCs --->
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: [2175:0x5e3a2b0] 173240537 ms: Mark-sweep 1997.3 (2096.0) -> 1981.5 (2096.0) MB, 2238.8 / 0.1 ms (average mu = 0.705, current mu = 0.689) task scavenge might not succeed
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: [2175:0x5e3a2b0] 173249525 ms: Mark-sweep 1995.2 (2096.0) -> 1981.9 (2096.5) MB, 2718.3 / 0.2 ms (average mu = 0.701, current mu = 0.698) task scavenge might not succeed
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: <--- JS stacktrace --->
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: FATAL ERROR: Reached heap limit Allocation failed - JavaScript heap out of memory
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 1: 0xb02ec0 node::Abort() [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 2: 0xa181fb node::FatalError(char const*, char const*) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 3: 0xced88e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 4: 0xcedc07 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 5: 0xea5ea5 [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 6: 0xeb557d v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 7: 0xf18844 v8::internal::ScavengeJob::Task::RunInternal() [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 8: 0xdabcfb non-virtual thunk to v8::internal::CancelableTask::Run() [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 9: 0xb707a4 node::PerIsolatePlatformData::RunForegroundTask(std::unique_ptr<v8::Task, std::default_delete<v8::Task> >) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 10: 0xb72609 node::PerIsolatePlatformData::FlushForegroundTasksInternal() [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 11: 0x155a396 [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 12: 0x156c804 [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 13: 0x155acc8 uv_run [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 14: 0xa3ffe5 node::SpinEventLoop(node::Environment*) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 15: 0xb45056 node::NodeMainInstance::Run(node::EnvSerializeInfo const*) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 16: 0xac67e2 node::Start(int, char**) [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 17: 0x7f7dacd9b555 __libc_start_main [/lib64/libc.so.6]
Jun 28 14:11:03 elk-k3-kibana kibana[2175]: 18: 0xa3bfbc [/usr/share/kibana/bin/../node/bin/node]
Jun 28 14:11:04 elk-k3-kibana systemd[1]: kibana.service: main process exited, code=killed, status=6/ABRT
Jun 28 14:11:04 elk-k3-kibana systemd[1]: Unit kibana.service entered failed state.
Jun 28 14:11:04 elk-k3-kibana systemd[1]: kibana.service failed.
Jun 28 14:11:07 elk-k3-kibana systemd[1]: kibana.service holdoff time over, scheduling restart.
Jun 28 14:11:07 elk-k3-kibana systemd[1]: Stopped Kibana.
Jun 28 14:11:07 elk-k3-kibana systemd[1]: Started Kibana.
Jun 28 14:23:33 elk-k3-kibana kibana[25054]: FATAL Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT step after 15 attempts, terminating.
Jun 28 14:23:33 elk-k3-kibana systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Jun 28 14:23:33 elk-k3-kibana systemd[1]: Unit kibana.service entered failed state.
Jun 28 14:23:33 elk-k3-kibana systemd[1]: kibana.service failed.
Jun 28 14:23:36 elk-k3-kibana systemd[1]: kibana.service holdoff time over, scheduling restart.
Jun 28 14:23:36 elk-k3-kibana systemd[1]: Stopped Kibana.
Jun 28 14:23:36 elk-k3-kibana systemd[1]: Started Kibana.
Jun 28 14:36:02 elk-k3-kibana kibana[25167]: FATAL Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT step after 15 attempts, terminating.
Jun 28 14:36:02 elk-k3-kibana systemd[1]: kibana.service: main process exited, code=exited, status=1/FAILURE
Jun 28 14:36:02 elk-k3-kibana systemd[1]: Unit kibana.service entered failed state.
Jun 28 14:36:02 elk-k3-kibana systemd[1]: kibana.service failed.
Jun 28 14:36:05 elk-k3-kibana systemd[1]: kibana.service holdoff time over, scheduling restart.
Jun 28 14:36:05 elk-k3-kibana systemd[1]: Stopped Kibana.
Jun 28 14:36:05 elk-k3-kibana systemd[1]: Started Kibana.
I wanted to know why there was an OOM and why Kibana is failing to start after stopping and starting because of the error
FATAL Error: Unable to complete saved object migrations for the [.kibana] index: Unable to complete the OUTDATED_DOCUMENTS_SEARCH_CLOSE_PIT step after 15 attempts, terminating.
)