Multiline: flush downstream error java.io.IOException since upgrading from 6.8 to 7.12

[soulless]

We have a multiline config that we use for capturing exceptions.

codec => multiline {
    pattern => "^%{TIMESTAMP_ISO8601}"
    negate => true
    what => "previous"
}

It appears to still be working, but since upgrading from logstash 6.8.10 to 7.12.1, we get the following error repeatedly:

 Multiline: flush downstream error {:exception=>#<IOError: java.io.IOException: data to be written is bigger than page capacity>}

We use all the default settings in logstash. How can we resolve this error?

Thanks

[Badger]

@soulless

Multiline: flush downstream error {:exception=>#<IOError: java.io.IOException: data to be written is bigger than page capacity>}

This error is telling you that the persistent queue is trying to persist an event that is larger than the queue.page_capacity. The default value for that is 64 Mb. The default max_bytes for a multiline codec is 10 Mb, so it would appear you have reduced the page_capacity. Increase it.

[soulless]

Thanks, but I don't think that is it as we are using all the default configuration for logstash so the page size is 64Mb.

I also don't see where the large data to be written are coming from. I am tailing the log file it is using as its input and it has nothing that would trigger the multiline, but I see the error log repeatedly in the logstash log.

[Badger]

I don't know what to say. The multiline codec is getting the error when it tries to flush the events. Reading the code, the queue will always write an element to it unless the element is larger than the page capacity.