# Docker Logs keep getting dropped with tried to parse field \[image\] as object, but found a concrete value error

**URL:** <https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245>\
**Category:** Elastic Agent\
**Created:** [February 23, 2023, 3:01am UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245 "2023-02-23T03:01:16Z")\
**Posts on this page:** 11\
**Page:** 3

<div class="post-metadata">

**Author:** ![xeraa](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/xeraa/32/48181_2.png) [@xeraa](https://discuss.elastic.co/u/xeraa)\
**Post date:** [March 6, 2023, 3:35pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/41 "2023-03-06T15:35:23Z")

</div>

I moved the issue to [Docker integration no longer collects logs due to "object mapping for [container.image] tried to parse field [image] as object, but found a concrete value" error · Issue #5450 · elastic/integrations · GitHub](https://github.com/elastic/integrations/issues/5450)

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 6, 2023, 3:47pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/42 "2023-03-06T15:47:38Z")

</div>

The internal team is looking at it, they see the issue with no ETA on a fix yet.

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [March 6, 2023, 7:08pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/43 "2023-03-06T19:08:02Z")

</div>

> [@stephenb](#):
>
> ```auto
> ignore_missing: false
> fail_on_error: true
> 
> ```

Maybe we could relax those constraints?

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 6, 2023, 7:17pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/44 "2023-03-06T19:17:31Z")

</div>

@Luca_Belluccini

Yes I thought I fixed that I did on my side. I just didn't post it back.

You can see in the image that I did actually change it. I'll fix the text as well

Done fixed ✅

---

<div class="post-metadata">

**Author:** ![Luca\_Belluccini](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/luca_belluccini/32/33239_2.png) [@Luca\_Belluccini](https://discuss.elastic.co/u/Luca_Belluccini)\
**Post date:** [March 6, 2023, 8:11pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/45 "2023-03-06T20:11:25Z")

</div>

Sorry my bad I've not checked ☺  
Thank you for sharing the workaround!

---

<div class="post-metadata">

**Author:** ![Raphasle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raphasle/32/118166_2.png) [@Raphasle](https://discuss.elastic.co/u/Raphasle)\
**Post date:** [March 7, 2023, 3:40pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/46 "2023-03-07T15:40:08Z")

</div>

After implementing the temporary workaround, I kept getting a further somewhat similar error causing (some of) the events to be dropped:

> failed to parse field [event.dataset] of type [constant\_keyword] in document with id 'MTqjvIYBCKefJtjV4yIM'. Preview of field's value: 'elasticsearch.server'","caused\_by":{"type":"illegal\_argument\_exception","reason":"[constant\_keyword] field [event.dataset] only accepts values that are equal to the value defined in the mappings [docker.container\_logs], but got [elasticsearch.server]"}

So I extended the workaround by another processor to just drop this field:

```auto
- drop_fields:
    fields: ["event.dataset"]
    ignore_missing: true

```

Maybe that's related and can be fixed at the same time. My context is that I run fleet server, kibana and elasticsearch in a containers and the container logs are picked up by en elastic agent with the Docker integration using an ndjson parser (and now a rename and drop\_fields processor).

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 7, 2023, 6:05pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/47 "2023-03-07T18:05:54Z")

</div>

Hi @Raphasle Welcome to the community

Interesting.... can you show a sample log line that has that issue?

I would probably _ **not** _ drop the `event.dataset` field it is a pretty important field.

I would probably just set it to the correct value when not the correct value something like this... you will need to check the syntax (I did not get a chance)

`add_fields` will overwrite

```auto
- add_fields:
    when.not.equals.event.dataset: "docker.container_logs"
    target: ''
    fields:
      event.dataset: "docker.container_logs"

```

---

<div class="post-metadata">

**Author:** ![Raphasle](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/raphasle/32/118166_2.png) [@Raphasle](https://discuss.elastic.co/u/Raphasle)\
**Post date:** [March 7, 2023, 10:49pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/48 "2023-03-07T22:49:08Z")

</div>

Indeed, setting the field to the anyways constant value is nicer.

One of the elasticsearch docker log statements that caused such an error was:

> {"log":"{"@timestamp":"2023-03-07T17:59:59.196Z", "log.level": "WARN", "data\_stream.dataset":"deprecation.elasticsearch","data\_stream.namespace":"default","data\_stream.type":"logs","elasticsearch.elastic\_product\_origin":"kibana","elasticsearch.event.category":"api","elasticsearch.http.request.x\_opaque\_id":"8121dfe1-4417-44ff-b8e5-9c796ca3c145;kibana:application:management:","event.code":"open\_system\_index\_access","message":"this request accesses system indices: [.async-search, .security-7, .security-profile-8, .tasks, .transform-internal-005, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default" , "ecs.version": "1.2.0","service.name":"ES\_ECS","event.dataset":"deprecation.elasticsearch","process.thread.name":"elasticsearch[b4ed9f9676e7][transport\_worker][T#1]","log.logger":"org.elasticsearch.deprecation.cluster.metadata.IndexNameExpressionResolver","trace.id":"07162c3ff1e0f665772161f75ac8ce83","elasticsearch.cluster.uuid":"JTty1lfFSl2mwlDH4uX7kg","elasticsearch.node.id":"iVX4NNmxTjmun9VLOab2YA","elasticsearch.node.name":"b4ed9f9676e7","elasticsearch.cluster.name":"Test"}\n","stream":"stdout","time":"2023-03-07T17:59:59.197403077Z"}

The full "message" showing up then in a .ds-logs-elastic\_agent.filebeat-default index was:

> **Expand**
>
> Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Date(2023, time.March, 7, 17, 59, 59, 196000000, time.UTC), Meta:{"input\_id":"filestream-docker-0af2355f-95f6-4b98-a06b-614cb1c73d82-docker-b4ed9f9676e7c4961f59c341b2cd106ed06e75278b46e1ad925a4ef1fdf6e894","raw\_index":"logs-docker.container\_logs-default","stream\_id":"docker-container-logs-es01-b4ed9f9676e7c4961f59c341b2cd106ed06e75278b46e1ad925a4ef1fdf6e894"}, Fields:{"agent":{"ephemeral\_id":"a4445584-0e54-4798-b796-f0adbabcb011","id":"6fcc8131-3506-4af9-90f8-054700a299f4","name":"some-hostname","type":"filebeat","version":"8.6.0"},"container":{"id":"b4ed9f9676e7c4961f59c341b2cd106ed06e75278b46e1ad925a4ef1fdf6e894","image":{"name":"some.private.registry/elasticsearch:8.6.2"},"labels":{"com\_docker\_compose\_config-hash":"e9757ec0a983b15dfd096bdb863a692a27c05fde28a17861c540ebfd7fab1dd9","com\_docker\_compose\_container-number":"1","com\_docker\_compose\_depends\_on":"","com\_docker\_compose\_image":"sha256:04485c81cc2d3ae1ae68e8be87f8a5043d71496332680a5b2e58664be95461c5","com\_docker\_compose\_oneoff":"False","com\_docker\_compose\_project":"octopus","com\_docker\_compose\_project\_config\_files":"/usr/local/bin/docker-files/octopus/docker-compose.yml","com\_docker\_compose\_project\_environment\_file":"/usr/local/bin/docker-files/octopus/octopus.env","com\_docker\_compose\_project\_working\_dir":"/usr/local/bin/docker-files/octopus","com\_docker\_compose\_replace":"0a5d7f7238b908b667f6f012a2be92978ab997791b677af599f8086636269079","com\_docker\_compose\_service":"es01","com\_docker\_compose\_version":"2.16.0","org\_label-schema\_build-date":"2023-02-13T09:35:20.314882762Z","org\_label-schema\_license":"Elastic-License-2.0","org\_label-schema\_name":"Elasticsearch","org\_label-schema\_schema-version":"1.0","org\_label-schema\_url":"https://www.elastic.co/products/elasticsearch","org\_label-schema\_usage":"https://www.elastic.co/guide/en/elasticsearch/reference/index.html","org\_label-schema\_vcs-ref":"2d58d0f136141f03239816a4e360a8d17b6d8f29","org\_label-schema\_vcs-url":"https://github.com/elastic/elasticsearch","org\_label-schema\_vendor":"Elastic","org\_label-schema\_version":"8.6.2","org\_opencontainers\_image\_created":"2023-02-13T09:35:20.314882762Z","org\_opencontainers\_image\_documentation":"https://www.elastic.co/guide/en/elasticsearch/reference/index.html","org\_opencontainers\_image\_licenses":"Elastic-License-2.0","org\_opencontainers\_image\_ref\_name":"ubuntu","org\_opencontainers\_image\_revision":"2d58d0f136141f03239816a4e360a8d17b6d8f29","org\_opencontainers\_image\_source":"https://github.com/elastic/elasticsearch","org\_opencontainers\_image\_title":"Elasticsearch","org\_opencontainers\_image\_url":"https://www.elastic.co/products/elasticsearch","org\_opencontainers\_image\_vendor":"Elastic","org\_opencontainers\_image\_version":"8.6.2"},"name":"es01"},"data\_stream":{"dataset":"docker.container\_logs","namespace":"default","type":"logs"},"data\_stream.dataset":"deprecation.elasticsearch","data\_stream.namespace":"default","data\_stream.type":"logs","ecs":{"version":"8.0.0"},"ecs.version":"1.2.0","elastic\_agent":{"id":"6fcc8131-3506-4af9-90f8-054700a299f4","snapshot":false,"version":"8.6.0"},"elasticsearch.cluster.name":"Test","elasticsearch.cluster.uuid":"JTty1lfFSl2mwlDH4uX7kg","elasticsearch.elastic\_product\_origin":"kibana","elasticsearch.event.category":"api","elasticsearch.http.request.x\_opaque\_id":"8121dfe1-4417-44ff-b8e5-9c796ca3c145;kibana:application:management:","elasticsearch.node.id":"iVX4NNmxTjmun9VLOab2YA","elasticsearch.node.name":"b4ed9f9676e7","event":{"dataset":"docker.container\_logs"},"event.code":"open\_system\_index\_access","host":{"architecture":"x86\_64","containerized":false,"hostname":"some-hostname","id":"e8a3e697270844118813786bfb68962c","ip":["many IPs"],"mac":["many mac addresses"],"name":"some-hostname","os":{"codename":"buster","family":"debian","kernel":"4.19.0-23-amd64","name":"Debian GNU/Linux","platform":"debian","type":"linux","version":"10 (buster)"}},"input":{"type":"filestream"},"log":{"file":{"path":"/var/lib/docker/containers/b4ed9f9676e7c4961f59c341b2cd106ed06e75278b46e1ad925a4ef1fdf6e894/b4ed9f9676e7c4961f59c341b2cd106ed06e75278b46e1ad925a4ef1fdf6e894-json.log"},"offset":2288689},"log.level":"WARN","log.logger":"org.elasticsearch.deprecation.cluster.metadata.IndexNameExpressionResolver","message":"this request accesses system indices: [.async-search, .security-7, .security-profile-8, .tasks, .transform-internal-005, .transform-internal-007], but in a future major version, direct access to system indices will be prevented by default","process.thread.name":"elasticsearch[b4ed9f9676e7][transport\_worker][T#1]","service.name":"ES\_ECS","stream":"stdout","trace.id":"07162c3ff1e0f665772161f75ac8ce83"}, Private:(\*input\_logfile.updateOp)(0xc0017a9e30), TimeSeries:false}, Flags:0x1, Cache:publisher.EventCache{m:mapstr.M(nil)}} (status=400): {"type":"mapper\_parsing\_exception","reason":"failed to parse field [data\_stream.dataset] of type [constant\_keyword] in document with id 'Wdk7vYYBcPMgqNc3Mkbf'. Preview of field's value: 'deprecation.elasticsearch'","caused\_by":{"type":"illegal\_argument\_exception","reason":"[constant\_keyword] field [data\_stream.dataset] only accepts values that are equal to the value defined in the mappings [docker.container\_logs], but got [deprecation.elasticsearch]"}}, dropping event!

But I am sure it's not related to this particular log messgage, e.g. rolling over an index causes 15 log messages that get dropped due to a mismatch of event.dataset.

---

<div class="post-metadata">

**Author:** ![fabien9402](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/fabien9402/32/116740_2.png) [@fabien9402](https://discuss.elastic.co/u/fabien9402)\
**Post date:** [March 21, 2023, 3:14pm UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/49 "2023-03-21T15:14:02Z")

</div>

I have the same problem after upgrade to 8.6.2.

The container logs was drop because the strucutre of the log is :

```auto
container.image : "nameOfImage"

```

And the integration seems to need :

```auto
container.image : 
    name : "nameOfImage"

```

So I used this script in the pipeline :

```auto
def temp = ctx.container.image;
ctx.container.image = [];
ctx.container.image.name = temp;

```

But it seems to have a very high impact on the performance of my cluster.  
The CPU is alway at 100% used and CPU credit is empty.

Do you think it's because all logs should be refactored by the cluster when is received ?

Do you think it will be better if I used the "rename" processor instead of the modification in the pipeline by using "script" ?

Thanks,

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [March 22, 2023, 1:15am UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/50 "2023-03-22T01:15:41Z")

</div>

> [@fabien9402](#):
>
> So I used this script in the pipeline :
> 
> ```auto
> def temp = ctx.container.image;
> ctx.container.image = [];
> ctx.container.image.name = temp;
> 
> ```

I would always use simple processors over a script when you can...

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 19, 2023, 1:16am UTC](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245/51 "2023-04-19T01:16:15Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/docker-logs-keep-getting-dropped-with-tried-to-parse-field-image-as-object-but-found-a-concrete-value-error/326245.md?page=2)