Some more info on this
I tried increasing the memory for node.js.
export NODE_OPTIONS="--max-old-space-size=5000"
Still no luck. However using top command shows that node is using quite a bit of resources.
And then there was a crash.
[myserver ~]$ sudo /usr/share/kibana/bin/kibana --verbose -c /etc/kibana/kibana.yml
<--- Last few GCs --->
[30204:0x38bdb90] 890429 ms: Mark-sweep 1309.1 (1453.2) -> 1309.1 (1454.2) MB, 1509.8 / 0.0 ms allocation failure GC in old space requested
[30204:0x38bdb90] 891989 ms: Mark-sweep 1309.1 (1454.2) -> 1309.1 (1416.2) MB, 1560.4 / 0.0 ms last resort GC in old space requested
[30204:0x38bdb90] 893555 ms: Mark-sweep 1309.1 (1416.2) -> 1309.1 (1416.2) MB, 1566.2 / 0.0 ms last resort GC in old space requested
<--- JS stacktrace --->
==== JS stack trace =========================================
Security context: 0x28d66ae25879 <JSObject>
0: builtin exit frame: lastIndexOf(this=0x16726838f3c1 <Very long string[4569717]>,0x1d5dd118ca51 <String[1]\: \n>)
1: has_nlb(aka has_nlb) [0xcbd32e022d1 <undefined>:5970] [bytecode=0x1c7895855f29 offset=15](this=0xcbd32e022d1 <undefined>)
2: /* anonymous */(aka /* anonymous */) [0xcbd32e022d1 <undefined>:6070] [bytecode=0x3235f1f5d819 offset=60](this=0xcbd32e022d1 <undefined>,c=0x...
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - JavaScript heap out of memory
1: node::Abort() [/usr/share/kibana/bin/../node/bin/node]
2: 0x8c21ec [/usr/share/kibana/bin/../node/bin/node]
3: v8::Utils::ReportOOMFailure(char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
4: v8::internal::V8::FatalProcessOutOfMemory(char const*, bool) [/usr/share/kibana/bin/../node/bin/node]
5: v8::internal::Factory::NewRawTwoByteString(int, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
6: v8::internal::String::SlowFlatten(v8::internal::Handle<v8::internal::ConsString>, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
7: v8::internal::String::Flatten(v8::internal::Handle<v8::internal::String>, v8::internal::PretenureFlag) [/usr/share/kibana/bin/../node/bin/node]
8: v8::internal::String::LastIndexOf(v8::internal::Isolate*, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>, v8::internal::Handle<v8::internal::Object>) [/usr/share/kibana/bin/../node/bin/node]
9: v8::internal::Builtin_StringPrototypeLastIndexOf(int, v8::internal::Object**, v8::internal::Isolate*) [/usr/share/kibana/bin/../node/bin/node]
10: 0x527c5a0697d
Aborted
Weird as it may sound, looks like progress. Any pointers?