I got this strange (I've not seen it before) error from Filebeat running as a daemonset on Kubernetes (minikube) today. It might be a misconfiguration, that's why I'm posting here first.
Filebeat yaml:
filebeat.yml: |-
logging.level: info
filebeat.autodiscover:
providers:
- type: kubernetes
hints.enabled: true
include_annotations:
- "laas.express-rest.no/service"
- "laas.express-rest.no/type"
- "laas.express-rest.no/token"
- "co.elastic.logs/*"
- "co.elastic.logs/disable"
- "co.elastic.logs.nginx/disable"
processors:
- add_kubernetes_metadata:
in_cluster: true
- drop_fields:
fields: ["host"]
# This ensures that every log that passes has required fields
- drop_event:
when:
not.has_fields: ['kubernetes.annotations.laas.express-rest.no/service', 'kubernetes.annotations.laas.express-rest.no/token', 'kubernetes.annotations.laas.express-rest.no/type']
- rename:
fields:
- from: "kubernetes.annotations.laas.express-rest.no/service"
to: "_service"
- from: "kubernetes.annotations.laas.express-rest.no/token"
to: "_token"
- from: "kubernetes.annotations.laas.express-rest.no/type"
to: "type"
output.logstash:
hosts: ['${LOGSTASH_HOSTS}']
Some logs. Complete log file (debug logs, -v -d autodiscover, docker)
2018-10-29T14:21:46.315Z DEBUG [autodiscover] autodiscover/autodiscover.go:135 Reloading existing autodiscover configs after error
2018-10-29T14:21:46.315Z DEBUG [autodiscover] cfgfile/list.go:70 Starting reload procedure, current runners: 18
2018-10-29T14:21:46.315Z DEBUG [autodiscover] cfgfile/list.go:88 Start list: 1, Stop list: 0
2018-10-29T14:21:46.315Z WARN [cfgwarn] docker/input.go:46 EXPERIMENTAL: Docker input is enabled.
2018-10-29T14:21:46.315Z ERROR [autodiscover] cfgfile/list.go:104 Error creating runner from config: reading docker input config: missing field accessing 'containers.ids.0'
2018-10-29T14:21:56.315Z DEBUG [autodiscover] autodiscover/autodiscover.go:135 Reloading existing autodiscover configs after error
2018-10-29T14:21:56.315Z DEBUG [autodiscover] cfgfile/list.go:70 Starting reload procedure, current runners: 18
2018-10-29T14:21:56.316Z DEBUG [autodiscover] cfgfile/list.go:88 Start list: 1, Stop list: 0
2018-10-29T14:21:56.316Z WARN [cfgwarn] docker/input.go:46 EXPERIMENTAL: Docker input is enabled.
2018-10-29T14:21:56.316Z ERROR [autodiscover] cfgfile/list.go:104 Error creating runner from config: reading docker input config: missing field accessing 'containers.ids.0'
2018-10-29T14:22:06.316Z DEBUG [autodiscover] autodiscover/autodiscover.go:135 Reloading existing autodiscover configs after error
2018-10-29T14:22:06.316Z DEBUG [autodiscover] cfgfile/list.go:70 Starting reload procedure, current runners: 18
2018-10-29T14:22:06.316Z DEBUG [autodiscover] cfgfile/list.go:88 Start list: 1, Stop list: 0
2018-10-29T14:22:06.316Z WARN [cfgwarn] docker/input.go:46 EXPERIMENTAL: Docker input is enabled.
2018-10-29T14:22:06.316Z ERROR [autodiscover] cfgfile/list.go:104 Error creating runner from config: reading docker input config: missing field accessing 'containers.ids.0'
2018-10-29T14:22:09.979Z INFO [monitoring] log/log.go:141 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":23850,"time":{"ms":21}},"total":{"ticks":36210,"time":{"ms":189},"value":36210},"user":{"ticks":12360,"time":{"ms":168}}},"info":{"ephemeral_id":"68c73467-10fb-491d-b82f-fdc41991d459","uptime":{"ms":1170137}},"memstats":{"gc_next":9865024,"memory_alloc":6600632,"memory_total":479523192}},"filebeat":{"events":{"active":1,"added":35,"done":34},"harvester":{"open_files":10,"running":8}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":30,"batches":15,"total":30},"read":{"bytes":90},"write":{"bytes":11160}},"pipeline":{"clients":48,"events":{"active":1,"filtered":4,"published":31,"total":35},"queue":{"acked":30}}},"registrar":{"states":{"current":17,"update":34},"writes":{"success":19,"total":19}},"system":{"load":{"1":1.81,"15":1.24,"5":1.36,"norm":{"1":0.905,"15":0.62,"5":0.68}}}}}}
2018-10-29T14:22:16.317Z DEBUG [autodiscover] autodiscover/autodiscover.go:135 Reloading existing autodiscover configs after error
2018-10-29T14:22:16.317Z DEBUG [autodiscover] cfgfile/list.go:70 Starting reload procedure, current runners: 18
2018-10-29T14:22:16.317Z DEBUG [autodiscover] cfgfile/list.go:88 Start list: 1, Stop list: 0
2018-10-29T14:22:16.317Z WARN [cfgwarn] docker/input.go:46 EXPERIMENTAL: Docker input is enabled.
2018-10-29T14:22:16.317Z ERROR [autodiscover] cfgfile/list.go:104 Error creating runner from config: reading docker input config: missing field accessing 'containers.ids.0'
2018-10-29T14:22:26.317Z DEBUG [autodiscover] autodiscover/autodiscover.go:135 Reloading existing autodiscover configs after error
2018-10-29T14:22:26.317Z DEBUG [autodiscover] cfgfile/list.go:70 Starting reload procedure, current runners: 18
2018-10-29T14:22:26.317Z DEBUG [autodiscover] cfgfile/list.go:88 Start list: 1, Stop list: 0
2018-10-29T14:22:26.317Z WARN [cfgwarn] docker/input.go:46 EXPERIMENTAL: Docker input is enabled.
2018-10-29T14:22:26.318Z ERROR [autodiscover] cfgfile/list.go:104 Error creating runner from config: reading docker input config: missing field accessing 'containers.ids.0'
2018-10-29T14:22:36.319Z DEBUG [autodiscover] autodiscover/autodiscover.go:135 Reloading existing autodiscover configs after error
2018-10-29T14:22:36.319Z DEBUG [autodiscover] cfgfile/list.go:70 Starting reload procedure, current runners: 18
2018-10-29T14:22:36.319Z DEBUG [autodiscover] cfgfile/list.go:88 Start list: 1, Stop list: 0
2018-10-29T14:22:36.319Z WARN [cfgwarn] docker/input.go:46 EXPERIMENTAL: Docker input is enabled.
2018-10-29T14:22:36.319Z ERROR [autodiscover] cfgfile/list.go:104 Error creating runner from config: reading docker input config: missing field accessing 'containers.ids.0'
