I have a new ELK server running on RHEL 6 that I am starting to ramp up sending application log files to (approx 7Gb/day so far). I found that my Kibana server died yesterday with the following error in stderr:
chroot: cannot change root directory to : No such file or directory
chroot: cannot change root directory to : No such file or directory
Debug: internal, implementation, error
Error: ENOSPC, no space left on device
at Error (native)
at Object.fs.writeSync (fs.js:613:20)
at SyncWriteStream.write (fs.js:1856:6)
at KbnLoggerJsonFormat.ondata (_stream_readable.js:540:20)
at KbnLoggerJsonFormat.emit (events.js:107:17)
at readableAddChunk (_stream_readable.js:163:16)
at KbnLoggerJsonFormat.Readable.push (_stream_readable.js:126:10)
at KbnLoggerJsonFormat.Transform.push (_stream_transform.js:140:32)
at KbnLoggerJsonFormat._transform (/opt/kibana/src/server/logging/LogFormat.js:61:12)
at KbnLoggerJsonFormat.Transform._read (_stream_transform.js:179:10)
at KbnLoggerJsonFormat.Transform._write (_stream_transform.js:167:12)
at doWrite (_stream_writable.js:301:12)
at writeOrBuffer (_stream_writable.js:288:5)
at KbnLoggerJsonFormat.Writable.write (_stream_writable.js:217:11)
at [object Object].ondata (_stream_readable.js:540:20)
at [object Object].emit (events.js:107:17)
FATAL ERROR: CALL_AND_RETRY_LAST Allocation failed - process out of memory
I don't see any evidence of swapping or high CPU on the machine a this time. I do see where an attempt to write a coredump to /var failed due to lack of space in that filesystem. ( I have expanded /var )
snippet from /var/log/messages:
Feb 17 09:49:01 dsa528 abrt[2446]: Write error: No space left on device
Feb 17 09:49:02 dsa528 abrt[2446]: Error writing '/var/spool/abrt/ccpp-2016-02-17-09:48:47-59476.new/coredump'
Are there any tuning suggestions or other things I should be watching to determine what caused this?