Out-of-memory issues in 7.6

[sherzinger (Sascha Herzinger)]

Hello,

we're having some issues starting Kibana 7.6.2.
During the optimization step this error is being thrown.

We have a VM with 16GB RAM and 8 vCPU with this setting in the Dockerfile:
ENV NODE_OPTIONS="--max-old-space-size=8192"

We also went into the container and confirmed that this env variable is actually set.

We tried less and more than 8G, with the same result.
For the ES instance we went as low as 1GB, just to free as much of the 16GB as possible. (We also cleared page cache, just to make sure)

We do not experience this issue in 7.2.1.

Any ideas on how to fix this are appreciated.

    kibana    | FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
    kibana    |  1: 0x8fa090 node::Abort() [/usr/share/kibana/bin/../node/bin/node]
    kibana    |  2: 0x8fa0dc  [/usr/share/kibana/bin/../node/bin/node]
    kibana    | 
    kibana    | <--- Last few GCs --->
    kibana    | 
    kibana    | [6:0x2b1e110]    73743 ms: Scavenge 1344.6 (1422.1) -> 1343.8 (1422.6) MB, 2.1 / 0.0 ms  (average mu = 0.205, current mu = 0.152) allocation failure 
    kibana    | [6:0x2b1e110]    73750 ms: Scavenge 1344.7 (1422.6) -> 1343.9 (1423.1) MB, 2.2 / 0.0 ms  (average mu = 0.205, current mu = 0.152) allocation failure 
    kibana    | [6:0x2b1e110]    73756 ms: Scavenge 1344.8 (1423.1) -> 1344.0 (1423.6) MB, 2.4 / 0.0 ms  (average mu = 0.205, current mu = 0.152) allocation failure 
    kibana    | 
    kibana    | 
    kibana    | <--- JS stacktrace --->
    kibana    | 
    kibana    | ==== JS stack trace =========================================
    kibana    | 
    kibana    |     0: ExitFrame [pc: 0x18a842c5be1d]
    kibana    | Security context: 0x34a12719e6e9 <JSObject>
    kibana    |     1: 0x37e2a5f07a31 <Symbol: Symbol.split>(aka [Symbol.split]) [0x34a127189641](this=0x1376f3c15471 <JSRegExp <String[5]: \r?\n>>,0x1376f3c153c9 <String[17]: _defineProperties>,0x37e2a5f026f1 <undefined>)
    kibana    |     2: split [0x34a1271906c9](this=0x1376f3c153c9 <String[17]: _defineProperties>,0x1376f3c15471 <JSRegExp <String[5]: \r?\n>>)
    kibana    |     3: w(aka w) [0x7499b8...
    kibana    | 
    kibana    |  3: 0xb0052e v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
    kibana    |  4: 0xb00764 v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
    kibana    |  5: 0xef4c72  [/usr/share/kibana/bin/../node/bin/node]
    kibana    |  6: 0xef4d78 v8::internal::Heap::CheckIneffectiveMarkCompact(unsigned long, double) [/usr/share/kibana/bin/../node/bin/node]
    kibana    |  7: 0xf00e52 v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/share/kibana/bin/../node/bin/node]
    kibana    |  8: 0xf01784 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/share/kibana/bin/../node/bin/node]
    kibana    |  9: 0xf043f1 v8::internal::Heap::AllocateRawWithRetryOrFail(int, v8::internal::AllocationSpace, v8::internal::AllocationAlignment) [/usr/share/kibana/bin/../node/bin/node]
    kibana    | 10: 0xecd4e6 v8::internal::Factory::AllocateRawArray(int, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
    kibana    | 11: 0xecdd6a v8::internal::Factory::NewFixedArrayWithFiller(v8::internal::Heap::RootListIndex, int, v8::internal::Object*, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
    kibana    | 12: 0xece2f7 v8::internal::Factory::NewFixedArrayWithHoles(int, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
    kibana    | 13: 0x11a78ab v8::internal::Runtime_RegExpSplit(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/bin/../node/bin/node]
    kibana    | 14: 0x18a842c5be1d 

[Mikhail_Shustov]

--max-old-space-size=8192 should suffice, we run Kibana 4Gb in dev mode (yes, it's a lot and we are working to reduce this number).
Do you have any custom plugins installed? Could you try to disable them in kibana.yml file with {pluginName}.enabled: false? Could you also disable some unnecessary Kibana plugins?
https://www.elastic.co/guide/en/kibana/current/disable-plugin.html

[sherzinger (Sascha Herzinger)]

Okay, I got a tip from somewhere else to modify the bin/kibana script in the docker image instead of passing this option as a env variable.

NODE_OPTIONS="--no-warnings --max-http-header-size=65536 ${NODE_OPTIONS}" NODE_ENV=production exec "${NODE}" --max-old-space-size=4096 "${DIR}/src/cli" ${@}

This works. Without the --max-old-space-size=4096 this fails, despite env showing me NODE_OPTIONS=--max-old-space-size=4096

Is that a bug in node.js maybe? This looks like environment variables a treated different from parameters.