Metricbeat: Cannot index event publisher.Event for org_opencontainers_image_created

[jmcclure (James McClure)]

Elasticsearch 7.9.3 (OSS)
Kibana 7.9.3 (OSS)
Metricbeat 7.9.3

Hi,

I have the Metricbeat docker module enabled with a number of metricsets.

---
- module: docker
  metricsets:
    - container
    - info
    - cpu
    - diskio
    - memory
    - network
    - healthcheck
...

When the module performs its collection cycle and publishes to Elasticsearch, I am seeing a warning in the Metricbeat log with the following reason:

failed to parse date field [2020-08-09 00:00:00+01:00] with format [strict_date_optional_time||epoch_millis]

The date value is coming from the docker container label from within the Kibana container.

"org_opencontainers_image_created": "2020-08-09 00:00:00+01:00"

"docker": {
    "container": {
      "labels": {
        "org_label-schema_name": "kibana",
        "org_label-schema_version": "7.9.3",
        "license": "ASL 2.0",
        "org_label-schema_license": "ASL 2.0",
        "description": "Kibana OSS 7.9.3",
        "org_label-schema_usage": "https://www.elastic.co/guide/en/kibana/index.html",
        "org_opencontainers_image_vendor": "CentOS",
        "org_label-schema_url": "https://www.elastic.co/products/kibana",
        "org_label-schema_vcs-url": "https://github.com/elastic/kibana",
        "org_label-schema_schema-version": "1.0",
        "org_opencontainers_image_licenses": "GPL-2.0-only",
        "org_opencontainers_image_created": "2020-08-09 00:00:00+01:00",
        "org_label-schema_vendor": "Elastic",
        "org_opencontainers_image_title": "CentOS Base Image",
        "org_label-schema_build-date": "2020-10-16T11:42:35.701Z",
        "maintainer": "NETSCOUT"
      }
    },

Log event:

2021-05-04T23:03:08.785Z        WARN    [elasticsearch] elasticsearch/client.go:408     Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0xc01c928a550aac74, ext:95363499716, loc:(*time.Location)(0x5803300)}, Meta:null, Fields:{"agent":{"ephemeral_id":"52b87b27-2929-4496-91cb-7edb728f5304","hostname":"xxxxxxxx","id":"8f9ba47e-03da-4c75-9cf6-6756ca727b30","name":"metricbeat","type":"metricbeat","version":"7.9.3"},"container":{"id":"c9408ef88ea219f4f0ad975ac9f78a04d0346f396f809a01c783cb79a0d91560","image":{"name":"xxxxxxxx:5600/kibana:7.9.3"},"name":"kibana","runtime":"docker"},"docker":{"container":{"labels":{"description":"Kibana OSS 7.9.3","license":"ASL 2.0","maintainer":"xxxxxxxx","org_label-schema_build-date":"2020-10-16T11:42:35.701Z","org_label-schema_license":"ASL 2.0","org_label-schema_name":"kibana","org_label-schema_schema-version":"1.0","org_label-schema_url":"https://www.elastic.co/products/kibana","org_label-schema_usage":"https://www.elastic.co/guide/en/kibana/index.html","org_label-schema_vcs-url":"https://github.com/elastic/kibana","org_label-schema_vendor":"Elastic","org_label-schema_version":"7.9.3","org_opencontainers_image_created":"2020-08-09 00:00:00+01:00","org_opencontainers_image_licenses":"GPL-2.0-only","org_opencontainers_image_title":"CentOS Base Image","org_opencontainers_image_vendor":"CentOS"}},"diskio":{"read":{"bytes":0,"ops":0,"queued":0,"rate":0.000000,"service_time":0,"wait_time":0},"reads":0.000000,"summary":{"bytes":984098304,"ops":80712,"queued":0,"rate":0.000000,"service_time":0,"wait_time":0},"total":0.000000,"write":{"bytes":984098304,"ops":80712,"queued":0,"rate":0.000000,"service_time":0,"wait_time":0},"writes":0.000000}},"ecs":{"version":"1.5.0"},"event":{"dataset":"docker.diskio","duration":2252880446,"module":"docker"},"host":{"name":"metricbeat"},"hostname":"dub-nbadev2-mw3.labs.netscout.com","metricset":{"name":"diskio","period":60000},"service":{"address":"/var/run/docker.sock","type":"docker"},"stats":"docker","tags":["docker","dub-nbadev2-mw3.labs.netscout.com"]}, Private:interface {}(nil), TimeSeries:true}, Flags:0x0, Cache:publisher.EventCache{m:common.MapStr(nil)}} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [docker.container.labels.org_opencontainers_image_created] of type [date] in document with id '3IOeOXkB62wddXGY8pPH'. Preview of field's value: '2020-08-09 00:00:00+01:00'","caused_by":{"type":"illegal_argument_exception","reason":"failed to parse date field [2020-08-09 00:00:00+01:00] with format [strict_date_optional_time||epoch_millis]","caused_by":{"type":"date_time_parse_exception","reason":"date_time_parse_exception: Failed to parse with all enclosed parsers"}}}

This occurs for all of the metric sets for the Kibana container.

Is there something that I can configure to allow this date format in addition to the other format variants that Elasticsearch can successfully consume? I believe that the Metricbeat index template specifies that the docker.container mapping is an 'object' type.

James

[ChrsMark (Chris Mark)]

Hi!

This is really strange, I don't see a reason of docker.container.labels.* being parsed in that way but I might miss some content here. Could you please open a GH issue for this?
Meanwhile you can use Drop fields from events | Metricbeat Reference [7.12] | Elastic processor so as to explicitly remove these fields and make it possible to index the events in ES.

[jmcclure (James McClure)]

@ChrsMark

Unfortunately while gathering information to open an issue for this, I am no longer able to reproduced the warning. So I will hold off on that until I see the occurrence again.

Marking your reply as a solution. I am just adding the drop field strategy to for the container labeling since I do not use it downstream anyway.

James