Filebeat Version:6.2.4
get decoding error when try to get docker logs
ERROR log/harvester.go:243 Read line error: decoding docker JSON: EOF; File: %!
logs are like this
{"log":"19:58:08.815 [http-nio-8080-exec-20] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : ] with root cause\n","stream":"stdout","time":"2018-06-21T11:58:08.815764952Z"}
{"log":"com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stdout","time":"2018-06-21T11:58:08.815849364Z"}
{"log":"Exception in thread \"pool-31-thread-33\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:10.625076429Z"}
{"log":"Exception in thread \"pool-31-thread-34\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:15.645079322Z"}
{"log":"Exception in thread \"pool-31-thread-35\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:20.664371726Z"}
steffens
June 22, 2018, 1:44pm
2
Can you share your filebeat config?
Did you configure log rotation of json files in docker?
I upgraded my filebeat to 6.3.0 and it seems to be resolved..
but now get a new error
2018-06-23T15:57:20.597+0800 ERROR log/harvester.go:255 Read line error: invalid CRI log; File: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
full debug log:
2018-06-23T15:57:20.596+0800 DEBUG [input] log/input.go:147 Start next scan
2018-06-23T15:57:20.596+0800 DEBUG [input] log/input.go:362 Check file for harvesting: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.596+0800 DEBUG [input] log/input.go:448 Update existing file for harvesting: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.596+0800 DEBUG [input] log/input.go:457 Resuming harvesting of file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0, new size: 1127762786
2018-06-23T15:57:20.596+0800 DEBUG [harvester] log/harvester.go:462 Setting offset for file based on seek: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.596+0800 DEBUG [harvester] log/harvester.go:448 Setting offset for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log. Offset: 0
2018-06-23T15:57:20.596+0800 DEBUG [harvester] log/harvester.go:363 Update state: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.596+0800 DEBUG [input] log/input.go:168 input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-23T15:57:20.597+0800 DEBUG [acker] beater/acker.go:47 stateful ack {"count": 1}
2018-06-23T15:57:20.597+0800 DEBUG [registrar] registrar/registrar.go:232 Processing 1 events
2018-06-23T15:57:20.597+0800 INFO log/harvester.go:228 Harvester started for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800 DEBUG [registrar] registrar/registrar.go:202 Registrar state updates processed. Count: 1
2018-06-23T15:57:20.597+0800 DEBUG [registrar] registrar/registrar.go:263 Write registry file: /var/lib/filebeat/registry
2018-06-23T15:57:20.597+0800 DEBUG [multiline] reader/multiline.go:126 Multiline event flushed because timeout reached.
2018-06-23T15:57:20.597+0800 ERROR log/harvester.go:255 Read line error: invalid CRI log; File: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800 DEBUG [harvester] log/harvester.go:483 Stopping harvester for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800 DEBUG [harvester] log/harvester.go:493 Closing file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800 DEBUG [harvester] log/harvester.go:363 Update state: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.597+0800 DEBUG [harvester] log/harvester.go:504 harvester cleanup finished for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
filebeat.yml docker type config:
- type: docker
multiline.match: after
multiline.negate: true
multiline.pattern: "^([0-9]+:?)+"
containers.ids:
- '6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b'
#json.ignore_decoding_error: true
#json.message_key: log
#json.keys_under_root: true
#json.add_error_key: true
processors:
- add_docker_metadata: ~
docker logs file:
head -n 100 /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
{"log":"2017-08-15T23:04:06.267 System is launching\n","stream":"stdout","time":"2017-08-15T15:04:06.269712026Z"}
{"log":"Go go go, Pikachu!\n","stream":"stdout","time":"2017-08-15T15:04:07.520748841Z"}
{"log":" `;-. ___, \n","stream":"stdout","time":"2017-08-15T15:04:07.520778302Z"}
{"log":" `.`\\_...._/`.-\"` \n","stream":"stdout","time":"2017-08-15T15:04:07.520783412Z"}
{"log":" \\ / , \n","stream":"stdout","time":"2017-08-15T15:04:07.520787846Z"}
{"log":" /() () \\ .' `-._ \n","stream":"stdout","time":"2017-08-15T15:04:07.520792301Z"}
{"log":" |) . ()\\ / _.' \n","stream":"stdout","time":"2017-08-15T15:04:07.520796464Z"}
{"log":" \\ -'- ,; '. \u003c \n","stream":"stdout","time":"2017-08-15T15:04:07.520800555Z"}
{"log":" ;.__ ,;| \u003e \\ \n","stream":"stdout","time":"2017-08-15T15:04:07.520805339Z"}
{"log":" / , / , |.-'.-' \n","stream":"stdout","time":"2017-08-15T15:04:07.520809487Z"}
{"log":" (_/ (_/ ,;|.\u003c` \n","stream":"stdout","time":"2017-08-15T15:04:07.520813518Z"}
{"log":" \\ , ;-` \n","stream":"stdout","time":"2017-08-15T15:04:07.52082107Z"}
{"log":" \u003e \\ / \n","stream":"stdout","time":"2017-08-15T15:04:07.520825243Z"}
{"log":" (_,-'`\u003e .' \n","stream":"stdout","time":"2017-08-15T15:04:07.520829355Z"}
{"log":" (_,' \n","stream":"stdout","time":"2017-08-15T15:04:07.520833406Z"}
{"log":"System initialization completed in 81384 ms\n","stream":"stdout","time":"2017-08-15T15:05:27.660512666Z"}
...
try to print message content
and got this...
system
July 21, 2018, 4:30pm
6
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
