Filebeat docker input: Error getting container info: <nil>

justinw · October 24, 2018, 9:56pm

Hi,

Running filebeat 6.3.2 with "- add_docker_metadata" set, and periodically see this in the logs:

ERROR docker/watcher.go:233 Error getting container info: <nil>

Can you please provide some information as to what may be causing this?

Thanks, Justin

pierhugues · October 25, 2018, 1:27pm

From looking at the code and your error trace:

github.com

elastic/beats/blob/99824df163863478d27688206e18ec27bc1b3ac5/libbeat/common/docker/watcher.go#L247-L253


					containers, err := w.listContainers(types.ContainerListOptions{
						Filters: filter,
					})
					if err != nil || len(containers) != 1 {
						logp.Err("Error getting container info: %v", err)
						continue
					}

It appears we make an API call and don't receive any containers information.

how often you see this in your log?

justinw · October 25, 2018, 5:55pm

Not super often, but often enough that I felt it was worth asking about.

Looking at that loop, it seems like the only way to get there is after a 'start' or 'update' event. Is it potentially a timing issue / race condition if the watcher got an event for a start or update but cannot find the container by id?

Best, Justin

justinw · October 25, 2018, 9:20pm

I also see this one, a little more frequently

2018-10-25T20:46:10.588Z ERROR kubernetes/watcher.go:254 kubernetes: Watching API error EOF

Looks like that comes from either of these two places:

github.com

elastic/beats/blob/master/libbeat/common/kubernetes/watcher.go#L244


			return
		default:
		}


		logp.Info("kubernetes: %s", "Watching API for resource events")


		watcher, err := w.client.Watch(w.ctx, w.options.Namespace, w.k8sResourceFactory(), w.buildOpts()...)
		if err != nil {
			//watch failures should be logged and gracefully failed over as metadata retrieval
			//should never stop.
			logp.Err("kubernetes: Watching API error %v", err)
			backoff(failures)
			failures++
			continue
		}


		for {
			r := w.k8sResourceFactory()
			eventType, err := watcher.Next(r)
			if err != nil {
				logp.Err("kubernetes: Watching API error %v", err)

github.com

elastic/beats/blob/master/libbeat/common/kubernetes/watcher.go#L254


			logp.Err("kubernetes: Watching API error %v", err)
			backoff(failures)
			failures++
			continue
		}


		for {
			r := w.k8sResourceFactory()
			eventType, err := watcher.Next(r)
			if err != nil {
				logp.Err("kubernetes: Watching API error %v", err)
				watcher.Close()
				if !(err == io.EOF || err == io.ErrUnexpectedEOF) {
					// This is an error event which can be recovered by moving to the latest resource version
					logp.Info("kubernetes: Ignoring event, moving to most recent resource version")
					w.lastResourceVersion = ""
				}
				break
			}
			failures = 0
			switch eventType {

Is this something to be concerned about?

system · November 22, 2018, 9:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Filebeat's docker metadata watcher losing container from its list somehow Beats filebeat	6	1087	December 29, 2017
Add docker metadata not working even after trying all possible combinations Beats filebeat	9	1193	December 25, 2018
Add_kubernetes_metadata intermittent failure Beats filebeat	1	223	June 2, 2023
Azure Kubernetes Services - metadata API error Beats filebeat	3	1030	September 25, 2018
Filebeat not reading all docker container logs Beats filebeat	11	5179	February 5, 2019

Filebeat docker input: Error getting container info: <nil>

Related topics