Kibana 4.3.1 died with process out of memory


I have a new ELK server running on RHEL 6 that I am starting to ramp up sending application log files to (approx 7Gb/day so far). I found that my Kibana server died yesterday with the following error in stderr:
chroot: cannot change root directory to : No such file or directory
chroot: cannot change root directory to : No such file or directory
Debug: internal, implementation, error
Error: ENOSPC, no space left on device
at Error (native)
at Object.fs.writeSync (fs.js:613:20)
at SyncWriteStream.write (fs.js:1856:6)
at KbnLoggerJsonFormat.ondata (_stream_readable.js:540:20)
at KbnLoggerJsonFormat.emit (events.js:107:17)
at readableAddChunk (_stream_readable.js:163:16)
at KbnLoggerJsonFormat.Readable.push (_stream_readable.js:126:10)
at KbnLoggerJsonFormat.Transform.push (_stream_transform.js:140:32)
at KbnLoggerJsonFormat._transform (/opt/kibana/src/server/logging/LogFormat.js:61:12)
at KbnLoggerJsonFormat.Transform._read (_stream_transform.js:179:10)
at KbnLoggerJsonFormat.Transform._write (_stream_transform.js:167:12)
at doWrite (_stream_writable.js:301:12)
at writeOrBuffer (_stream_writable.js:288:5)
at KbnLoggerJsonFormat.Writable.write (_stream_writable.js:217:11)
at [object Object].ondata (_stream_readable.js:540:20)
at [object Object].emit (events.js:107:17)
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory

I don't see any evidence of swapping or high CPU on the machine a this time. I do see where an attempt to write a coredump to /var failed due to lack of space in that filesystem. ( I have expanded /var )

snippet from /var/log/messages:
Feb 17 09:49:01 dsa528 abrt[2446]: Write error: No space left on device
Feb 17 09:49:02 dsa528 abrt[2446]: Error writing '/var/spool/abrt/'

Are there any tuning suggestions or other things I should be watching to determine what caused this?

(Michael) #2

see if setting --max-old-space-size= help in keeping kibana from running the server out of ram. Kibana's node.js leaks memory. see Kibana 4.2 server memory usage and the issue listed in the post on git.

(system) #3