# Filebeat decoding Docker logs error

**URL:** https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961
**Category:** Beats
**Tags:** filebeat
**Created:** [June 22, 2018, 2:36am UTC](https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961 "2018-06-22T02:36:33Z")
**Posts on this page:** 6
**Page:** 1

<div class="post-metadata">

### Author: ![d54c364cb1c1253c751d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d54c364cb1c1253c751d/32/32571_2.png) [@d54c364cb1c1253c751d](https://discuss.elastic.co/u/d54c364cb1c1253c751d)
#### Post date: [June 22, 2018, 2:36am UTC](https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961/1 "2018-06-22T02:36:33Z")

</div>

Filebeat Version:6.2.4  
Docker version: 1.12.5, build 7392c3b

get decoding error when try to get docker logs

```auto
 ERROR	log/harvester.go:243	Read line error: decoding docker JSON: EOF; File: %!

```

logs are like this

```auto
{"log":"19:58:08.815 [http-nio-8080-exec-20] ERROR o.a.c.c.C.[.[.[.[dispatcherServlet] - Servlet.service() for servlet [dispatcherServlet] in context with path [] threw exception [Request processing failed; nested exception is com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json :] with root cause\n","stream":"stdout","time":"2018-06-21T11:58:08.815764952Z"}
{"log":"com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stdout","time":"2018-06-21T11:58:08.815849364Z"}
{"log":"Exception in thread \"pool-31-thread-33\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:10.625076429Z"}
{"log":"Exception in thread \"pool-31-thread-34\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:15.645079322Z"}
{"log":"Exception in thread \"pool-31-thread-35\" com.alibaba.fastjson.JSONException: syntax error, expect {, actual EOF, pos 0, json : \n","stream":"stderr","time":"2018-06-21T11:58:20.664371726Z"}

```

---

<div class="post-metadata">

### Author: ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)
#### Post date: [June 22, 2018, 1:44pm UTC](https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961/2 "2018-06-22T13:44:27Z")

</div>

Can you share your filebeat config?

Did you configure log rotation of json files in docker?

---

<div class="post-metadata">

### Author: ![d54c364cb1c1253c751d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d54c364cb1c1253c751d/32/32571_2.png) [@d54c364cb1c1253c751d](https://discuss.elastic.co/u/d54c364cb1c1253c751d)
#### Post date: [June 23, 2018, 1:55am UTC](https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961/3 "2018-06-23T01:55:58Z")

</div>

I upgraded my filebeat to 6.3.0 and it seems to be resolved..

but now get a new error

```auto
2018-06-23T15:57:20.597+0800	ERROR	log/harvester.go:255	Read line error: invalid CRI log; File: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log

```

full debug log:

```auto
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:147	Start next scan
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:362	Check file for harvesting: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:448	Update existing file for harvesting: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:457	Resuming harvesting of file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0, new size: 1127762786
2018-06-23T15:57:20.596+0800	DEBUG	[harvester]	log/harvester.go:462	Setting offset for file based on seek: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.596+0800	DEBUG	[harvester]	log/harvester.go:448	Setting offset for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log. Offset: 0
2018-06-23T15:57:20.596+0800	DEBUG	[harvester]	log/harvester.go:363	Update state: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.596+0800	DEBUG	[input]	log/input.go:168	input states cleaned up. Before: 1, After: 1, Pending: 0
2018-06-23T15:57:20.597+0800	DEBUG	[acker]	beater/acker.go:47	stateful ack	{"count": 1}
2018-06-23T15:57:20.597+0800	DEBUG	[registrar]	registrar/registrar.go:232	Processing 1 events
2018-06-23T15:57:20.597+0800	INFO	log/harvester.go:228	Harvester started for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[registrar]	registrar/registrar.go:202	Registrar state updates processed. Count: 1
2018-06-23T15:57:20.597+0800	DEBUG	[registrar]	registrar/registrar.go:263	Write registry file: /var/lib/filebeat/registry
2018-06-23T15:57:20.597+0800	DEBUG	[multiline]	reader/multiline.go:126	Multiline event flushed because timeout reached.
2018-06-23T15:57:20.597+0800	ERROR	log/harvester.go:255	Read line error: invalid CRI log; File: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:483	Stopping harvester for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:493	Closing file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:363	Update state: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log, offset: 0
2018-06-23T15:57:20.597+0800	DEBUG	[harvester]	log/harvester.go:504	harvester cleanup finished for file: /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log

```

filebeat.yml docker type config:

```auto
  - type: docker
    multiline.match: after
    multiline.negate: true
    multiline.pattern: "^([0-9]+:?)+"
    containers.ids:
      - '6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b'
    #json.ignore_decoding_error: true
    #json.message_key: log
    #json.keys_under_root: true
    #json.add_error_key: true
    processors:
      - add_docker_metadata: ~

```

---

<div class="post-metadata">

### Author: ![d54c364cb1c1253c751d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d54c364cb1c1253c751d/32/32571_2.png) [@d54c364cb1c1253c751d](https://discuss.elastic.co/u/d54c364cb1c1253c751d)
#### Post date: [June 23, 2018, 8:12am UTC](https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961/4 "2018-06-23T08:12:07Z")

</div>

docker logs file:

```auto
 head -n 100 /var/lib/docker/containers/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b/6edf6cc78b6fdc6df310394762777263e063df8bde28c70ae233027535ca9d6b-json.log

{"log":"2017-08-15T23:04:06.267 System is launching\n","stream":"stdout","time":"2017-08-15T15:04:06.269712026Z"}
{"log":"Go go go, Pikachu!\n","stream":"stdout","time":"2017-08-15T15:04:07.520748841Z"}
{"log":" `;-. ___, \n","stream":"stdout","time":"2017-08-15T15:04:07.520778302Z"}
{"log":" `.`\\_...._/`.-\"` \n","stream":"stdout","time":"2017-08-15T15:04:07.520783412Z"}
{"log":" \\ / , \n","stream":"stdout","time":"2017-08-15T15:04:07.520787846Z"}
{"log":" /() () \\ .' `-._ \n","stream":"stdout","time":"2017-08-15T15:04:07.520792301Z"}
{"log":" |) . ()\\ / _.' \n","stream":"stdout","time":"2017-08-15T15:04:07.520796464Z"}
{"log":" \\ -'- ,; '. \u003c \n","stream":"stdout","time":"2017-08-15T15:04:07.520800555Z"}
{"log":" ;.__ ,;| \u003e \\ \n","stream":"stdout","time":"2017-08-15T15:04:07.520805339Z"}
{"log":" / , / , |.-'.-' \n","stream":"stdout","time":"2017-08-15T15:04:07.520809487Z"}
{"log":" (_/ (_/ ,;|.\u003c` \n","stream":"stdout","time":"2017-08-15T15:04:07.520813518Z"}
{"log":" \\ , ;-` \n","stream":"stdout","time":"2017-08-15T15:04:07.52082107Z"}
{"log":" \u003e \\ / \n","stream":"stdout","time":"2017-08-15T15:04:07.520825243Z"}
{"log":" (_,-'`\u003e .' \n","stream":"stdout","time":"2017-08-15T15:04:07.520829355Z"}
{"log":" (_,' \n","stream":"stdout","time":"2017-08-15T15:04:07.520833406Z"}
{"log":"System initialization completed in 81384 ms\n","stream":"stdout","time":"2017-08-15T15:05:27.660512666Z"}
...

```

---

<div class="post-metadata">

### Author: ![d54c364cb1c1253c751d](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/d54c364cb1c1253c751d/32/32571_2.png) [@d54c364cb1c1253c751d](https://discuss.elastic.co/u/d54c364cb1c1253c751d)
#### Post date: [June 23, 2018, 4:29pm UTC](https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961/5 "2018-06-23T16:29:10Z")

</div>

try to print message content

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/8/b8f11327e4472d6ce2dde525965023720f8ff174.png)  
and got this...  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/e/fe15ed95902fc5c7c32921671c9f26a9dde34b95.jpg)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [July 21, 2018, 4:30pm UTC](https://discuss.elastic.co/t/filebeat-decoding-docker-logs-error/136961/6 "2018-07-21T16:30:17Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
