Filebeat Autodiscover gives fatal error: concurrent map iteration and map write

Alsheh · August 21, 2018, 6:35am

Using the official filebeat docker image form elastic, I get fatal error: concurrent map iteration and map write (detailed logs included below) when using the filebeat.autodiscover in the config file. However, I cannot reproduce this error with an uncontainerized version of filebeat even when the same config file is used!

filebeat.yml

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.image: '.*'
          config:
            - type: docker
              containers:
                path: '/usr/share/filebeat/containers/'
                stream: 'all'
                ids:
                  - '${data.docker.container.id}'
              processors:
                - add_docker_metadata: ~
                - add_host_metadata:
                    netinfo.enabled: true
output.logstash:
  hosts: ['logstash:5044', 'logstash2:5044']
  loadbalance: true

Error:

fatal error: concurrent map iteration and map write
goroutine 32 [running]:
runtime.throw(0x1638ce3, 0x26)
    /usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc42259b5f0 sp=0xc42259b5d0 pc=0xaba0e5
runtime.mapiternext(0xc42259b6c0)
    /usr/local/go/src/runtime/hashmap.go:778 +0x6f1 fp=0xc42259b688 sp=0xc42259b5f0 pc=0xa98291
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInterface(0xc4203f32c0, 0x14c5ac0, 0xc42074eb10, 0xc42074eb10, 0x16610d8)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map.go:34 +0xea fp=0xc42259b730 sp=0xc42259b688 pc=0xeb3b5a
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc4203f32c0, 0x15aa700, 0xc42074eb10, 0x0, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:89 +0x15b fp=0xc42259b7a8 sp=0xc42259b730 pc=0xeb1b6b
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInlineInterface(0xc4203f32c0, 0x15aa700, 0xc421b8e410, 0x95, 0x15aa700, 0xc421b8e410)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map_inline.generated.go:44 +0x158 fp=0xc42259b868 sp=0xc42259b7a8 pc=0xeb46e8
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldInlineFold.func1(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x0, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:292 +0x86 fp=0xc42259b8b8 sp=0xc42259b868 pc=0xf17e06
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x0, 0x15397c0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:177 +0x88 fp=0xc42259b910 sp=0xc42259b8b8 pc=0xf17c18
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeStructFold.func1(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x0, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:167 +0x95 fp=0xc42259b958 sp=0xc42259b910 pc=0xf17b15
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldAnyReflect(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x99, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:511 +0xb9 fp=0xc42259b998 sp=0xc42259b958 pc=0xeb9ce9
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0xc421b8e3c0, 0xc421b8e3c0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:92 +0x1c9 fp=0xc42259ba10 sp=0xc42259b998 pc=0xeb1bd9
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.(*Iterator).Fold(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0xc421b8e3c0, 0xc4208d3bb0)
  ...

Any help to get the containerized version to work would be appreciated!

UPDADE: this issue is now reproducible with an uncontainerzied filebeat.

pierhugues · August 21, 2018, 6:20pm

Looking at the panic message this looks like a concurrency issue, which is why its hard to reproduce.

Which version of the docker image are you running?

Alsheh · August 21, 2018, 6:32pm

Hi @pierhugues, I am using this filebeat image: docker.elastic.co/elasticsearch/elasticsearch:6.3.2. And, yes, you're right its hard to trigger this error on my end as well; I just leave filebeat running for few minutes to harvest logs for other images and it occurs at random times. FYI, the docker images for which I am harvesting the logs include Elasticsearch, Logstash, Kibana, and two other micro services. Also, all the elastic stack images have the same major and minor version.

P.S. the elastic stack and the other apps all exist on the same host and their logs, which are being collected by filebeat, are located at /usr/share/filebeat/containers/ for testing purposes.

pierhugues · August 21, 2018, 6:46pm

@Alsheh

I've created https://github.com/elastic/beats/issues/8040 to follow up the issue on our tracker,
I haven't reproduced the error locally, but the most plausible cause would be one of the processors used in the configuration is not copying a shared data structure and a goroutine is trying to write to it when we are in the process of serializing the data.

Can you remove the processor one by one and see if you still have the issue, this will help us narrow the problem.

pierhugues · August 21, 2018, 7:15pm

@Alsheh

You can remove the - add_docker_metadata: ~ this processor is not needed since the docker autodiscover will automatically add the metadata.

Alsheh · August 21, 2018, 8:05pm

@pierhugues thanks for taking interest in this issue. I let filebeat run with no processors and it was harvesting ~4k logs/30s but it crashed after 30 minutes

filebeat.yml:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.image: '.*'
          config:
            - type: docker
              containers:
                path: '/usr/share/filebeat/containers-logs/'
                stream: 'all'
                ids:
                  - '${data.docker.container.id}'
output.logstash:
  hosts: ['logstash:5044', 'logstash2:5044']
  loadbalance: true

Error message:

fatal error: concurrent map iteration and map write

goroutine 32 [running]:
runtime.throw(0x1638ce3, 0x26)
	/usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc4207c95f0 sp=0xc4207c95d0 pc=0xaba0e5
runtime.mapiternext(0xc4207c96c0)
	/usr/local/go/src/runtime/hashmap.go:778 +0x6f1 fp=0xc4207c9688 sp=0xc4207c95f0 pc=0xa98291
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInterface(0xc420440660, 0x14c5ac0, 0xc4200651d0, 0xc4200651d0, 0x16610d8)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map.go:34 +0xea fp=0xc4207c9730 sp=0xc4207c9688 pc=0xeb3b5a
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc420440660, 0x15aa700, 0xc4200651d0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:89 +0x15b fp=0xc4207c97a8 sp=0xc4207c9730 pc=0xeb1b6b
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInlineInterface(0xc420440660, 0x15aa700, 0xc4220fbdf0, 0x95, 0x15aa700, 0xc4220fbdf0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map_inline.generated.go:44 +0x158 fp=0xc4207c9868 sp=0xc4207c97a8 pc=0xeb46e8
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldInlineFold.func1(0xc420440660, 0x15397c0, 0xc4220fbda0, 0x99, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:292 +0x86 fp=0xc4207c98b8 sp=0xc4207c9868 pc=0xf17e06
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc420440660, 0x15397c0, 0xc4220fbda0, 0x99, 0x0, 0x15397c0)
...

goroutine 1 [chan receive, 20 minutes]:
github.com/elastic/beats/filebeat/beater.(*signalWait).Wait(...)
	/go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:28
github.com/elastic/beats/filebeat/beater.(*Filebeat).Run(0xc42043e9e0, 0xc42017c380, 0x0, 0x0)
	/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:389 +0xd2a
github.com/elastic/beats/libbeat/cmd/instance.(*Beat).launch(0xc42017c380, 0x1660868, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:321 +0x462
github.com/elastic/beats/libbeat/cmd/instance.Run.func1(0x161cada, 0x8, 0x161cada, 0x8, 0x0, 0x0, 0x1660868, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:146 +0xfa
github.com/elastic/beats/libbeat/cmd/instance.Run(0x161cada, 0x8, 0x161cada, 0x8, 0x0, 0x0, 0x1660868, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:147 +0x71
github.com/elastic/beats/libbeat/cmd.genRunCmd.func1(0xc4203aa280, 0xc420319d50, 0x0, 0x1)
	/go/src/github.com/elastic/beats/libbeat/cmd/run.go:19 +0x60
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).execute(0xc4203aa280, 0xc42000c070, 0x1, 0x1, 0xc4203aa280, 0xc42000c070)
	/go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:704 +0x2c6
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4203aa280, 0x7ffc35e70e00, 0xc4203c61e0, 0xc4203c65a0)
	/go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:785 +0x30e
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4203aa280, 0xc420225f70, 0x13c1309)
	/go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:738 +0x2b
main.main()
	/go/src/github.com/elastic/beats/filebeat/main.go:18 +0x2f
...

goroutine 9 [syscall, 20 minutes]:
os/signal.signal_recv(0x16622c0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
...

Alsheh · August 21, 2018, 8:26pm

@pierhugues
UPDADE: this issue is now reproducible with an uncontainerzied filebeat.

pierhugues · August 22, 2018, 12:35pm

@Alsheh using the same configuration?

Alsheh · August 23, 2018, 4:56am

@pierhugues
The containerized and uncontainerized filebeats were failing when I included the add_host_metadata processor in the configuration. After removing this processor, things seems to be working just fine. Consequently, I need to figure out an alternative way to collect the host meta data with filebeat. Any ideas?

Working filebeat.yml:

filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.image: '.*'
          config:
            - type: docker
              containers:
                path: '/usr/share/filebeat/containers/'
                stream: 'all'
                ids:
                  - '${data.docker.container.id}'
output.logstash:
  hosts: ['logstash:5044', 'logstash2:5044']
  loadbalance: true

Alsheh · August 30, 2018, 5:43pm

UPDATE: upgrading to filebeat 6.4.0 fixes the issue

system · September 27, 2018, 5:43pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.