Filebeat 5.0 with multiline, split event data to two events

Partial events being written normally happens due to buffering effects. E.g. if one is using a BufferedWriter in Java , the buffer will only be flushed once the buffer becomes full or the writer is closed (no buffer timeout). This is quite common behavior in many output implementations using buffering for high increasing throughput. This unfortunately increases the latencies in the logging case. If no more events are logged, events might partially being stuck in buffer for undetermined amount of time (or might get lost if application crashes).

Personally I consider this to be an issue in the application writing the logs. If something bad happens I'd prefer my logs to be flushed more 'nowish'.

Option is to set timeout to a very large value or to -1 (disable timeout). Problem with disabling timeout is, if last event is multiline event we can not tell if multiline event is finished until either another event is received or file is closed (again increasing latencies).