Kubernetes + Auto-discovery + dedot

Elastic Stack Beats
1

With Filebeat 6.8.0, where exactly do the labels.dedot and annotations.dedot options go in the Filebeat config when using autodiscovery?

Lack of dedotting is responsible for errors like:

failed to parse field [kubernetes.labels.app] of type [keyword]

Complete error:

2019-09-26T19:30:41.040Z	WARN	elasticsearch/client.go:539	Cannot index event publisher.Event{Content:beat.Event{Timestamp:time.Time{wall:0x19e536e8, ext:63705123033, loc:(*time.Location)(nil)}, Meta:common.MapStr(nil), Fields:common.MapStr{"input":common.MapStr{"type":"docker"}, "message":"time=\"2019-09-26T19:30:33Z\" level=info msg=\"finished unary call with code OK\" grpc.code=OK grpc.method=GenerateManifest grpc.request.deadline=\"2019-09-26T19:31:33Z\" grpc.service=repository.RepoServerService grpc.start_time=\"2019-09-26T19:30:33Z\" grpc.time_ms=83.238 span.kind=server system=grpc", "source":"/var/lib/docker/containers/6057d6c2e2adc4380965cfdbc4444c1351fa709d1365fb2e9608a137953ee06d/6057d6c2e2adc4380965cfdbc4444c1351fa709d1365fb2e9608a137953ee06d-json.log", "offset":7177467, "prospector":common.MapStr{"type":"docker"}, "kubernetes":common.MapStr{"pod":common.MapStr{"name":"argocd-repo-server-5f464b4d8f-ms5nb", "uid":"0df37a2b-9b6d-11e9-ab21-0e7653edd380"}, "node":common.MapStr{"name":"ip-10-10-101-95.ec2.internal"}, "container":common.MapStr{"name":"argocd-repo-server"}, "namespace":"infra", "replicaset":common.MapStr{"name":"argocd-repo-server-5f464b4d8f"}, "labels":common.MapStr{"app":common.MapStr{"kubernetes":common.MapStr{"io/name":"argocd-repo-server"}}, "pod-template-hash":"1902060849"}}, "beat":common.MapStr{"hostname":"filebeat-ksz6h", "version":"6.8.0", "name":"filebeat-ksz6h"}, "log":common.MapStr{"file":common.MapStr{"path":"/var/lib/docker/containers/6057d6c2e2adc4380965cfdbc4444c1351fa709d1365fb2e9608a137953ee06d/6057d6c2e2adc4380965cfdbc4444c1351fa709d1365fb2e9608a137953ee06d-json.log"}}, "stream":"stderr", "host":common.MapStr{"name":"filebeat-ksz6h"}, "meta":common.MapStr{"cloud":common.MapStr{"instance_id":"i-003208b1c30cd527e", "machine_type":"t3.medium", "region":"us-east-1", "availability_zone":"us-east-1b", "provider":"ec2"}}}, Private:file.State{Id:"", Finished:false, Fileinfo:(*os.fileStat)(0xc420954680), Source:"/var/lib/docker/containers/6057d6c2e2adc4380965cfdbc4444c1351fa709d1365fb2e9608a137953ee06d/6057d6c2e2adc4380965cfdbc4444c1351fa709d1365fb2e9608a137953ee06d-json.log", Offset:7177838, Timestamp:time.Time{wall:0xbf5b61cfba659432, ext:70061440418, loc:(*time.Location)(0x21ec6a0)}, TTL:-1, Type:"docker", Meta:map[string]string(nil), FileStateOS:file.StateOS{Inode:0xbc10425, Device:0x10301}}}, Flags:0x1} (status=400): {"type":"mapper_parsing_exception","reason":"failed to parse field [kubernetes.labels.app] of type [keyword]","caused_by":{"type":"illegal_state_exception","reason":"Can't get text on a START_OBJECT at 1:295"}}

I have tried the following configs with no success:

In the add_kubernetes_metadata block per https://github.com/elastic/beats/issues/8773#issuecomment-478930844

filebeat.autodiscover:
  providers:
    - type: kubernetes
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_kubernetes_metadata:
      in_cluster: true
      annotations.dedot: true
      labels.dedot: true

At the top of the kubernetes provider block, which is where it supposedly goes for the docker provider (per https://www.elastic.co/guide/en/beats/filebeat/6.8/configuration-autodiscover.html):

filebeat.autodiscover:
  providers:
    - type: kubernetes
      labels.dedot: true
      annotations.dedot: true
      hints.enabled: true
      hints.default_config:
        type: container
        paths:
          - /var/log/containers/*${data.kubernetes.container.id}.log

processors:
  - add_cloud_metadata:
  - add_kubernetes_metadata:
      in_cluster: true

Can someone provide an example of a full, working Filebeat config with Kubernetes label dedotting enabled?

1 Like
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.