Filebeat docker input: Error getting container info: <nil>

justinw · October 24, 2018, 9:56pm

Hi,

Running filebeat 6.3.2 with "- add_docker_metadata" set, and periodically see this in the logs:

ERROR docker/watcher.go:233 Error getting container info: <nil>

Can you please provide some information as to what may be causing this?

Thanks, Justin

pierhugues · October 25, 2018, 1:27pm

From looking at the code and your error trace:

github.com

elastic/beats/blob/99824df163863478d27688206e18ec27bc1b3ac5/libbeat/common/docker/watcher.go#L247-L253


					containers, err := w.listContainers(types.ContainerListOptions{
						Filters: filter,
					})
					if err != nil || len(containers) != 1 {
						logp.Err("Error getting container info: %v", err)
						continue
					}

It appears we make an API call and don't receive any containers information.

how often you see this in your log?

justinw · October 25, 2018, 5:55pm

Not super often, but often enough that I felt it was worth asking about.

Looking at that loop, it seems like the only way to get there is after a 'start' or 'update' event. Is it potentially a timing issue / race condition if the watcher got an event for a start or update but cannot find the container by id?

Best, Justin

justinw · October 25, 2018, 9:20pm

I also see this one, a little more frequently

2018-10-25T20:46:10.588Z ERROR kubernetes/watcher.go:254 kubernetes: Watching API error EOF

Looks like that comes from either of these two places:

github.com

elastic/beats/blob/master/libbeat/common/kubernetes/watcher.go#L244


			return
		default:
		}


		logp.Info("kubernetes: %s", "Watching API for resource events")


		watcher, err := w.client.Watch(w.ctx, w.options.Namespace, w.k8sResourceFactory(), w.buildOpts()...)
		if err != nil {
			//watch failures should be logged and gracefully failed over as metadata retrieval
			//should never stop.
			logp.Err("kubernetes: Watching API error %v", err)
			backoff(failures)
			failures++
			continue
		}


		for {
			r := w.k8sResourceFactory()
			eventType, err := watcher.Next(r)
			if err != nil {
				logp.Err("kubernetes: Watching API error %v", err)

github.com

elastic/beats/blob/master/libbeat/common/kubernetes/watcher.go#L254


			logp.Err("kubernetes: Watching API error %v", err)
			backoff(failures)
			failures++
			continue
		}


		for {
			r := w.k8sResourceFactory()
			eventType, err := watcher.Next(r)
			if err != nil {
				logp.Err("kubernetes: Watching API error %v", err)
				watcher.Close()
				if !(err == io.EOF || err == io.ErrUnexpectedEOF) {
					// This is an error event which can be recovered by moving to the latest resource version
					logp.Info("kubernetes: Ignoring event, moving to most recent resource version")
					w.lastResourceVersion = ""
				}
				break
			}
			failures = 0
			switch eventType {

Is this something to be concerned about?

system · November 22, 2018, 9:20pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.