wklken
July 17, 2018, 2:56am
1
Hi,
Currently, I met a problem with the json log with max_bytes limit.
filebeat.yml
# 100KB
max_bytes: 102400
json.message_key: message
json.add_error_key: true
then I run it on k8s with memory limit: 200Mi
when one large JSON log: 1 single line, 50M, JSON format.
The filebeat will be killed by k8s because of the memory. And rescheduled, start, then be killed again.
I read the doc, and is the process order decode the JSON -> multiline -> max_bytes?
Is there any other way to avoid this, truncate the log into max_bytes before JSON decode?
thanks!
adrisr
July 17, 2018, 7:59am
2
Yes, it's like that
//
// limit -> (multiline -> timeout) -> strip_newline -> json -> encode -> line -> log_file
//
// Each reader on the left, contains the reader on the right and calls `Next()` to fetch more data.
// At the base of all readers the the log_file reader. That means in the data is flowing in the opposite direction:
//
// log_file -> line -> encode -> json -> strip_newline -> (timeout -> multiline) -> limit
//
// log_file implements io.Reader interface and encode reader is an adapter for io.Reader to
// reader.Reader also handling file encodings. All other readers implement reader.Reader
func (h *Harvester) newLogFileReader() (reader.Reader, error) {
var r reader.Reader
var err error
// TODO: NewLineReader uses additional buffering to deal with encoding and testing
// for new lines in input stream. Simple 8-bit based encodings, or plain
// don't require 'complicated' logic.
h.log, err = NewLog(h.source, h.config.LogConfig)
if err != nil {
return nil, err
}
You can create an enhancement request with this problem and see if we can come up with a better solution.
wklken
July 17, 2018, 9:00am
3
Thank you! It helps a lot!
wklken
July 27, 2018, 3:13am
5
I have added few codes to support the feature maxLineBytes limit. (the value of maxLineBytes from config is not added yet)
I have already done some test manually, and put that into our testing environment.
I'm wondering if that a common issue, should I take more time to finish it and make a pr
system
August 24, 2018, 5:13am
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.