Filebeat decoding Docker logs error


(shuxiong) #1

Filebeat Version:6.2.4
Docker version: 1.12.5, build 7392c3b

get decoding error when try to get docker logs

 ERROR	log/harvester.go:243	Read line error: decoding docker JSON: EOF; File: %!

logs are like this

{"log":"19:58:08.815 [http-nio-8080-exec-20] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : ] with root cause\n","stream":"stdout","time":"2018-06-21T11:58:08.815764952Z"}
{"log":"com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stdout","time":"2018-06-21T11:58:08.815849364Z"}
{"log":"Exception in thread \"pool-31-thread-33\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:10.625076429Z"}
{"log":"Exception in thread \"pool-31-thread-34\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:15.645079322Z"}
{"log":"Exception in thread \"pool-31-thread-35\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:20.664371726Z"}

(Steffen Siering) #2

Can you share your filebeat config?

Did you configure log rotation of json files in docker?


(shuxiong) #3

I upgraded my filebeat to 6.3.0 and it seems to be resolved..

but now get a new error

2018-06-23T15:57:20.597+0800	ERROR	log/harvester.go:255	Read line error: invalid CRI log; File: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log

full debug log:

2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:147	Start next scan
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:362	Check file for harvesting: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:448	Update existing file for harvesting: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:457	Resuming harvesting of file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0, new size: 1127762786
2018-06-23T15:57:20.596+0800	DEBUG	[harvester]	log/harvester.go:462	Setting offset for file based on seek: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.596+0800	DEBUG	[harvester]	log/harvester.go:448	Setting offset for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log. Offset: 0
2018-06-23T15:57:20.596+0800	DEBUG	[harvester]	log/harvester.go:363	Update state: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:168	input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-23T15:57:20.597+0800	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-06-23T15:57:20.597+0800	DEBUG	[registrar]	registrar/registrar.go:232	Processing 1 events
2018-06-23T15:57:20.597+0800	INFO	log/harvester.go:228	Harvester started for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[registrar]	registrar/registrar.go:202	Registrar state updates processed. Count: 1
2018-06-23T15:57:20.597+0800	DEBUG	[registrar]	registrar/registrar.go:263	Write registry file: /var/lib/filebeat/registry
2018-06-23T15:57:20.597+0800	DEBUG	[multiline]	reader/multiline.go:126	Multiline event flushed because timeout reached.
2018-06-23T15:57:20.597+0800	ERROR	log/harvester.go:255	Read line error: invalid CRI log; File: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:483	Stopping harvester for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:493	Closing file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:363	Update state: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:504	harvester cleanup finished for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log

filebeat.yml docker type config:

  - type: docker
    multiline.match: after
    multiline.negate: true
    multiline.pattern: "^([0-9]+:?)+"
    containers.ids:
      - '6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b'
    #json.ignore_decoding_error: true
    #json.message_key: log
    #json.keys_under_root: true
    #json.add_error_key: true
    processors:
      - add_docker_metadata: ~

(shuxiong) #4

docker logs file:

 head -n 100 /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log

{"log":"2017-08-15T23:04:06.267 System is launching\n","stream":"stdout","time":"2017-08-15T15:04:06.269712026Z"}
{"log":"Go go go, Pikachu!\n","stream":"stdout","time":"2017-08-15T15:04:07.520748841Z"}
{"log":"     `;-.          ___,  \n","stream":"stdout","time":"2017-08-15T15:04:07.520778302Z"}
{"log":"       `.`\\_...._/`.-\"`  \n","stream":"stdout","time":"2017-08-15T15:04:07.520783412Z"}
{"log":"         \\        /      ,  \n","stream":"stdout","time":"2017-08-15T15:04:07.520787846Z"}
{"log":"         /()   () \\    .' `-._  \n","stream":"stdout","time":"2017-08-15T15:04:07.520792301Z"}
{"log":"        |)  .    ()\\  /   _.'  \n","stream":"stdout","time":"2017-08-15T15:04:07.520796464Z"}
{"log":"        \\  -'-     ,; '. \u003c  \n","stream":"stdout","time":"2017-08-15T15:04:07.520800555Z"}
{"log":"         ;.__     ,;|   \u003e \\  \n","stream":"stdout","time":"2017-08-15T15:04:07.520805339Z"}
{"log":"        / ,    / ,  |.-'.-'  \n","stream":"stdout","time":"2017-08-15T15:04:07.520809487Z"}
{"log":"       (_/    (_/ ,;|.\u003c`  \n","stream":"stdout","time":"2017-08-15T15:04:07.520813518Z"}
{"log":"         \\    ,     ;-`  \n","stream":"stdout","time":"2017-08-15T15:04:07.52082107Z"}
{"log":"          \u003e   \\    /  \n","stream":"stdout","time":"2017-08-15T15:04:07.520825243Z"}
{"log":"         (_,-'`\u003e .'  \n","stream":"stdout","time":"2017-08-15T15:04:07.520829355Z"}
{"log":"              (_,'  \n","stream":"stdout","time":"2017-08-15T15:04:07.520833406Z"}
{"log":"System initialization completed in 81384 ms\n","stream":"stdout","time":"2017-08-15T15:05:27.660512666Z"}
...

(shuxiong) #5

try to print message content


and got this...


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.