Edit the container log as per the objective for verification # vim /var/lib/docker/containers/d401788ed2c1186afed103ff05e0f9426e1a2657d541b84e3873fb1121ca9c6c/d401788ed2c1186afed103ff05e0f9426e1a2657d541b84e3873fb1121ca9c6c-json.log
{"log":"Hello from Docker! \n","stream":"stdout","time":"2019-03-15T08:46:59.401128771Z"}
{"stream":"stdout","time":"2019-03-15T08:46:59.401161003Z"}
[Note: a) 1st line is in correct format
b) In 2nd line "log" json field is missing
]
Start filebeat # filebeat -e
Observation:
It is observed that when container's log does not have "log" json field in log line but have "stream" and "time" fields, then filebeat can't parse the corresponding log line and exited immediately with below go routine panic error.
panic: runtime error: index out of range
Expected Behavior:
When "log" field is missing in any container's log line, then filebeat should treat it as "" and it should be treated as partial line by docker json parser during event preparation. This blank line will be combined with next line where log message have "\n" at the end just like the behavior with other partial lines.
Otherwise if somehow in any container's log the log data is corrupted then due to go panic error filebeat will be stopped working which will impact the production environment.
So I think enhancement is required in this area
Important Note:
I have found some code level enhancement. I verified it locally. After getting confirmation I will raise a bug and propose code level changes accordingly through PR.
Thank you for the summary! Could you please provide more information on when Docker emits a log line without the key "log"? Could you please share examples for partial lines?
I agree with you, Filebeat should not panic when the key is missing.
Basically till now I have not found the practical example when docker emits log line without the key "log". I have found this thing during POC / testing.
Basically as "autodiscover" is disabled so filebeat treats the containers log file as normal logs but as prospector type is set to "docker" so it will parse the log line through docker-json parser.
So If any one try to interrupt the production environment then they can corrupt it by sending a log line just like docker json log format without "log" key. So for the security purpose and to prevent failure we need to enhance the code to handle this kind of situation.
Example of Partial Line:
According to current behavior of filebeat docker parser, if json lines are as per below
line-1: "log": "This is first line"
line-2: "log":"This is second line \n"
line-3: "log": "This is third line \n"
During event preparation 2 events will be prepared
In 1st event : "message" : "This is first line This is second line"
In 2nd event: "message": "This is third line"
As line-1 has no "\n" so it is treated as partial line hence it is combined with line-2 during event preparation.
In mentioned scenario when "log" key is not present then the value of "log" key will be taken as "" and it should be combined with next complete line.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
