Docker input via autodiscovery not always json-decoding message


(Gunter Grodotzki) #1

using the docker-input it seems that sometimes the message is a json:

logstash_1_7c18eb226755 |        "message" => "{\"log\":\"192.168.80.1 - - [22/Nov/2018:10:38:46 +0000]  \\\"GET / HTTP/1.1\\\" 200 612 \\\"-\\\" \\\"curl/7.47.0\\\" \\\"localhost:8181\\\"\\n\",\"stream\":\"stdout\",\"time\":\"2018-11-22T10:38:46.945703331Z\"}",

and sometimes the message is json decoded:

logstash_1_f7e7e988a4af |       "@version" => "1",
logstash_1_f7e7e988a4af |         "offset" => 0,
logstash_1_f7e7e988a4af |         "stream" => "stdout",
logstash_1_f7e7e988a4af |           "host" => {
logstash_1_f7e7e988a4af |         "name" => "f3135fa546a2"
logstash_1_f7e7e988a4af |     },
logstash_1_f7e7e988a4af |         "source" => "/var/lib/docker/containers/c5b44c22a654d4622af2ad0df971d3eee35db318cccaf88ea9fb1a877882c108/c5b44c22a654d4622af2ad0df971d3eee35db318cccaf88ea9fb1a877882c108-json.log",
logstash_1_f7e7e988a4af |        "message" => "172.26.0.1 - - [22/Nov/2018:10:16:04 +0000] \"GET / HTTP/1.1\" 200 612 \"-\" \"curl/7.47.0\" \"-\""
logstash_1_f7e7e988a4af | }

(Gunter Grodotzki) #2

actually this is also happening when running filebeat natively (via official apt repo).


(Pier-Hugues Pellerin) #3

A few questions, which version are you running?

Is there any JSON parsing error on the Filebeat side?


(Gunter Grodotzki) #4

Right now it seems the autodiscovery plugin for docker is faulty. You can easily replicate by running filebeat as docker container and even natively.

There is no JSON error, but sometimes filebeat will json_decode the message field and sometimes not. As shown in my example above.

It ties to this problem as well: Docker + Persistent Registry = autodiscover failing

Once I switch away from autodiscovery everything works perfectly.


(Pier-Hugues Pellerin) #5

@lifeofguenter Thanks, I will let @exekias handle the issue in the other thread, he know everything about that part :slight_smile: