This weekend i had a situation where my logfile got somehow corrupted having binary data at the beginning (0x00), and there normal txt data appended to it. And filebeat was scanning this file to put it into ES.
It turned out, that filebeat was using 20GB of RAM (RES) and tried to put it to ES. ES of course responded, that it's too large, but still filebeat used enormous amount of RAM.
Then, i started looking for a way to limit a log line size (for sanity reasons), and unfortunatelly did not found any way to limit this in filebeat.yml config file.
Am I missing something in configuration? If not, then perhaps it would be a good idea to introduce such a configurable parameter to avoid situation i faced. Not to mention, that crafted long line could lead to a server DOS because of filebeat eating more than available memory on that server ath the moment.