# Filebeat Autodiscover gives fatal error: concurrent map iteration and map write

**URL:** <https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [August 21, 2018, 6:35am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296 "2018-08-21T06:35:18Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Alsheh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alsheh/32/36799_2.png) [@Alsheh](https://discuss.elastic.co/u/Alsheh)\
**Post date:** [August 21, 2018, 6:35am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/1 "2018-08-21T06:35:18Z")

</div>

Using the official filebeat docker image form elastic, I get `fatal error: concurrent map iteration and map write` (detailed logs included below) when using the `filebeat.autodiscover` in the config file. However, I cannot reproduce this error with an uncontainerized version of filebeat even when the same config file is used!

filebeat.yml

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.image: '.*'
          config:
            - type: docker
              containers:
                path: '/usr/share/filebeat/containers/'
                stream: 'all'
                ids:
                  - '${data.docker.container.id}'
              processors:
                - add_docker_metadata: ~
                - add_host_metadata:
                    netinfo.enabled: true
output.logstash:
  hosts: ['logstash:5044', 'logstash2:5044']
  loadbalance: true

```

Error:

```auto
fatal error: concurrent map iteration and map write
goroutine 32 [running]:
runtime.throw(0x1638ce3, 0x26)
    /usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc42259b5f0 sp=0xc42259b5d0 pc=0xaba0e5
runtime.mapiternext(0xc42259b6c0)
    /usr/local/go/src/runtime/hashmap.go:778 +0x6f1 fp=0xc42259b688 sp=0xc42259b5f0 pc=0xa98291
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInterface(0xc4203f32c0, 0x14c5ac0, 0xc42074eb10, 0xc42074eb10, 0x16610d8)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map.go:34 +0xea fp=0xc42259b730 sp=0xc42259b688 pc=0xeb3b5a
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc4203f32c0, 0x15aa700, 0xc42074eb10, 0x0, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:89 +0x15b fp=0xc42259b7a8 sp=0xc42259b730 pc=0xeb1b6b
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInlineInterface(0xc4203f32c0, 0x15aa700, 0xc421b8e410, 0x95, 0x15aa700, 0xc421b8e410)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map_inline.generated.go:44 +0x158 fp=0xc42259b868 sp=0xc42259b7a8 pc=0xeb46e8
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldInlineFold.func1(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x0, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:292 +0x86 fp=0xc42259b8b8 sp=0xc42259b868 pc=0xf17e06
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x0, 0x15397c0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:177 +0x88 fp=0xc42259b910 sp=0xc42259b8b8 pc=0xf17c18
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeStructFold.func1(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x0, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:167 +0x95 fp=0xc42259b958 sp=0xc42259b910 pc=0xf17b15
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldAnyReflect(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0x99, 0x99, 0x0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:511 +0xb9 fp=0xc42259b998 sp=0xc42259b958 pc=0xeb9ce9
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0xc421b8e3c0, 0xc421b8e3c0)
    /go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:92 +0x1c9 fp=0xc42259ba10 sp=0xc42259b998 pc=0xeb1bd9
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.(*Iterator).Fold(0xc4203f32c0, 0x15397c0, 0xc421b8e3c0, 0xc421b8e3c0, 0xc4208d3bb0)
  ...

```

Any help to get the containerized version to work would be appreciated!

* * *

**UPDADE** : this issue is now reproducible with an uncontainerzied filebeat.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 21, 2018, 6:20pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/2 "2018-08-21T18:20:02Z")

</div>

Looking at the panic message this looks like a concurrency issue, which is why its hard to reproduce.

Which version of the docker image are you running?

---

<div class="post-metadata">

**Author:** ![Alsheh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alsheh/32/36799_2.png) [@Alsheh](https://discuss.elastic.co/u/Alsheh)\
**Post date:** [August 21, 2018, 6:32pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/3 "2018-08-21T18:32:50Z")

</div>

Hi @pierhugues, I am using this filebeat image: `docker.elastic.co/elasticsearch/elasticsearch:6.3.2`. And, yes, you're right its hard to trigger this error on my end as well; I just leave filebeat running for few minutes to harvest logs for other images and it occurs at random times. FYI, the docker images for which I am harvesting the logs include Elasticsearch, Logstash, Kibana, and two other micro services. Also, all the elastic stack images have the same major and minor version.

P.S. the elastic stack and the other apps all exist on the same host and their logs, which are being collected by filebeat, are located at `/usr/share/filebeat/containers/` for testing purposes.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 21, 2018, 6:46pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/4 "2018-08-21T18:46:07Z")

</div>

@Alsheh

I've created [https://github.com/elastic/beats/issues/8040](https://github.com/elastic/beats/issues/8040) to follow up the issue on our tracker,  
I haven't reproduced the error locally, but the most plausible cause would be one of the processors used in the configuration is not copying a shared data structure and a goroutine is trying to write to it when we are in the process of serializing the data.

Can you remove the processor one by one and see if you still have the issue, this will help us narrow the problem.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 21, 2018, 7:15pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/5 "2018-08-21T19:15:45Z")

</div>

@Alsheh

You can remove the `- add_docker_metadata: ~` this processor is not needed since the docker autodiscover will automatically add the metadata.

---

<div class="post-metadata">

**Author:** ![Alsheh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alsheh/32/36799_2.png) [@Alsheh](https://discuss.elastic.co/u/Alsheh)\
**Post date:** [August 21, 2018, 8:05pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/6 "2018-08-21T20:05:45Z")

</div>

@pierhugues thanks for taking interest in this issue. I let filebeat run with no processors and it was harvesting ~4k logs/30s but it crashed after 30 minutes ☹

filebeat.yml:

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.image: '.*'
          config:
            - type: docker
              containers:
                path: '/usr/share/filebeat/containers-logs/'
                stream: 'all'
                ids:
                  - '${data.docker.container.id}'
output.logstash:
  hosts: ['logstash:5044', 'logstash2:5044']
  loadbalance: true

```

Error message:

```auto
fatal error: concurrent map iteration and map write

goroutine 32 [running]:
runtime.throw(0x1638ce3, 0x26)
	/usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc4207c95f0 sp=0xc4207c95d0 pc=0xaba0e5
runtime.mapiternext(0xc4207c96c0)
	/usr/local/go/src/runtime/hashmap.go:778 +0x6f1 fp=0xc4207c9688 sp=0xc4207c95f0 pc=0xa98291
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInterface(0xc420440660, 0x14c5ac0, 0xc4200651d0, 0xc4200651d0, 0x16610d8)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map.go:34 +0xea fp=0xc4207c9730 sp=0xc4207c9688 pc=0xeb3b5a
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc420440660, 0x15aa700, 0xc4200651d0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:89 +0x15b fp=0xc4207c97a8 sp=0xc4207c9730 pc=0xeb1b6b
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInlineInterface(0xc420440660, 0x15aa700, 0xc4220fbdf0, 0x95, 0x15aa700, 0xc4220fbdf0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map_inline.generated.go:44 +0x158 fp=0xc4207c9868 sp=0xc4207c97a8 pc=0xeb46e8
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldInlineFold.func1(0xc420440660, 0x15397c0, 0xc4220fbda0, 0x99, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:292 +0x86 fp=0xc4207c98b8 sp=0xc4207c9868 pc=0xf17e06
github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc420440660, 0x15397c0, 0xc4220fbda0, 0x99, 0x0, 0x15397c0)
...

goroutine 1 [chan receive, 20 minutes]:
github.com/elastic/beats/filebeat/beater.(*signalWait).Wait(...)
	/go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:28
github.com/elastic/beats/filebeat/beater.(*Filebeat).Run(0xc42043e9e0, 0xc42017c380, 0x0, 0x0)
	/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:389 +0xd2a
github.com/elastic/beats/libbeat/cmd/instance.(*Beat).launch(0xc42017c380, 0x1660868, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:321 +0x462
github.com/elastic/beats/libbeat/cmd/instance.Run.func1(0x161cada, 0x8, 0x161cada, 0x8, 0x0, 0x0, 0x1660868, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:146 +0xfa
github.com/elastic/beats/libbeat/cmd/instance.Run(0x161cada, 0x8, 0x161cada, 0x8, 0x0, 0x0, 0x1660868, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:147 +0x71
github.com/elastic/beats/libbeat/cmd.genRunCmd.func1(0xc4203aa280, 0xc420319d50, 0x0, 0x1)
	/go/src/github.com/elastic/beats/libbeat/cmd/run.go:19 +0x60
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).execute(0xc4203aa280, 0xc42000c070, 0x1, 0x1, 0xc4203aa280, 0xc42000c070)
	/go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:704 +0x2c6
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).ExecuteC(0xc4203aa280, 0x7ffc35e70e00, 0xc4203c61e0, 0xc4203c65a0)
	/go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:785 +0x30e
github.com/elastic/beats/vendor/github.com/spf13/cobra.(*Command).Execute(0xc4203aa280, 0xc420225f70, 0x13c1309)
	/go/src/github.com/elastic/beats/vendor/github.com/spf13/cobra/command.go:738 +0x2b
main.main()
	/go/src/github.com/elastic/beats/filebeat/main.go:18 +0x2f
...

goroutine 9 [syscall, 20 minutes]:
os/signal.signal_recv(0x16622c0)
	/usr/local/go/src/runtime/sigqueue.go:131 +0xa6
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x22
created by os/signal.init.0
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x41
...

```

---

<div class="post-metadata">

**Author:** ![Alsheh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alsheh/32/36799_2.png) [@Alsheh](https://discuss.elastic.co/u/Alsheh)\
**Post date:** [August 21, 2018, 8:26pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/7 "2018-08-21T20:26:08Z")

</div>

@pierhugues  
**UPDADE** : this issue is now reproducible with an uncontainerzied filebeat.

---

<div class="post-metadata">

**Author:** ![pierhugues](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/pierhugues/32/48383_2.png) [@pierhugues](https://discuss.elastic.co/u/pierhugues)\
**Post date:** [August 22, 2018, 12:35pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/8 "2018-08-22T12:35:12Z")

</div>

@Alsheh using the same configuration?

---

<div class="post-metadata">

**Author:** ![Alsheh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alsheh/32/36799_2.png) [@Alsheh](https://discuss.elastic.co/u/Alsheh)\
**Post date:** [August 23, 2018, 4:56am UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/9 "2018-08-23T04:56:18Z")

</div>

@pierhugues  
The containerized and uncontainerized filebeats were failing when I included the `add_host_metadata` processor in the configuration. After removing this processor, things seems to be working just fine. Consequently, I need to figure out an alternative way to collect the host meta data with filebeat. Any ideas?

Working filebeat.yml:

```auto
filebeat.autodiscover:
  providers:
    - type: docker
      templates:
        - condition:
            regexp:
              docker.container.image: '.*'
          config:
            - type: docker
              containers:
                path: '/usr/share/filebeat/containers/'
                stream: 'all'
                ids:
                  - '${data.docker.container.id}'
output.logstash:
  hosts: ['logstash:5044', 'logstash2:5044']
  loadbalance: true

```

---

<div class="post-metadata">

**Author:** ![Alsheh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alsheh/32/36799_2.png) [@Alsheh](https://discuss.elastic.co/u/Alsheh)\
**Post date:** [August 30, 2018, 5:43pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/10 "2018-08-30T17:43:46Z")

</div>

**UPDATE** : upgrading to filebeat 6.4.0 fixes the issue

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 27, 2018, 5:43pm UTC](https://discuss.elastic.co/t/filebeat-autodiscover-gives-fatal-error-concurrent-map-iteration-and-map-write/145296/11 "2018-09-27T17:43:51Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.