# Panic: fatal error: concurrent map iteration and map write

**URL:** <https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435>\
**Category:** Beats\
**Tags:** filebeat\
**Created:** [February 14, 2019, 3:07pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435 "2019-02-14T15:07:51Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![mkorolyov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mkorolyov/32/40936_2.png) [@mkorolyov](https://discuss.elastic.co/u/mkorolyov)\
**Post date:** [February 14, 2019, 3:07pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/1 "2019-02-14T15:07:51Z")

</div>

Hi,

We have separate filebeat per running container setup at the moment.  
Filebeat writing direct to elastic search.  
After upgrading from 5.6.14 to 6.4.2 filebeat version it started to crash after some time of up and running with

```
fatal error: concurrent map iteration and map write

    goroutine 60 [running]:
    runtime.throw(0x15dee00, 0x26)
           	/usr/local/go/src/runtime/panic.go:616 +0x81 fp=0xc420663640 sp=0xc420663620 pc=0x9b1ab1
    runtime.mapiternext(0xc420663708)
           	/usr/local/go/src/runtime/hashmap.go:747 +0x55c fp=0xc4206636d0 sp=0xc420663640 pc=0x98fa7c
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInterface(0xc420339470, 0x1444680, 0xc4203c0660, 0xc4203c0660, 0x160fbc0)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map.go:34 +0xea fp=0xc420663778 sp=0xc4206636d0 pc=0xda1aba
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc420339470, 0x1543160, 0xc4203c0660, 0x0, 0x0)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:89 +0x15b fp=0xc4206637f0 sp=0xc420663778 pc=0xd9fb4b
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldMapInlineInterface(0xc420339470, 0x1543160, 0xc42047b758, 0x95, 0x1543160, 0xc42047b758)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_map_inline.generated.go:44 +0x153 fp=0xc4206638a8 sp=0xc4206637f0 pc=0xda2643
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldInlineFold.func1(0xc420339470, 0x14a9d00, 0xc42047b740, 0x99, 0x0, 0x0)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:292 +0x86 fp=0xc4206638f8 sp=0xc4206638a8 pc=0xdf5d16
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeFieldsFold.func1(0xc420339470, 0x14a9d00, 0xc42047b740, 0x99, 0x0, 0x14a9d00)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:177 +0x88 fp=0xc420663950 sp=0xc4206638f8 pc=0xdf5b38
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.makeStructFold.func1(0xc420339470, 0x14a9d00, 0xc42047b740, 0x99, 0x0, 0x0)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:167 +0x95 fp=0xc420663998 sp=0xc420663950 pc=0xdf5a35
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldAnyReflect(0xc420339470, 0x14a9d00, 0xc42047b740, 0x99, 0x99, 0xc42005d000)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold_reflect.go:511 +0xb9 fp=0xc4206639d8 sp=0xc420663998 pc=0xda7799
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.foldInterfaceValue(0xc420339470, 0x14a9d00, 0xc42047b740, 0xc42047b740, 0xc42047b740)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:92 +0x1c9 fp=0xc420663a50 sp=0xc4206639d8 pc=0xd9fbb9
    github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype.(*Iterator).Fold(0xc420339470, 0x14a9d00, 0xc42047b740, 0xc42047b740, 0x0)
           	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-structform/gotype/fold.go:69 +0x41 fp=0xc420663a88 sp=0xc420663a50 pc=0xd9f9c1
    github.com/elastic/beats/libbeat/outputs/elasticsearch.(*jsonEncoder).AddRaw(0xc420332d60, 0x148db00, 0xc42054b200, 0x0, 0x0)
           	/go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/enc.go:119 +0x300 fp=0xc420663b88 sp=0xc420663a88 pc=0xe23d40
    github.com/elastic/beats/libbeat/outputs/elasticsearch.(*jsonEncoder).Add(0xc420332d60, 0x147c860, 0xc420193c00, 0x148db00, 0xc42054b200, 0xc420193c00, 0x0)
           	/go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/enc.go:139 +0x8b fp=0xc420663be8 sp=0xc420663b88 pc=0xe23dfb
    github.com/elastic/beats/libbeat/outputs/elasticsearch.bulkEncodePublishRequest(0x7fad51c7bbd0, 0xc420332d60, 0x1684760, 0xc420192240, 0x0, 0xc42054b080, 0x32, 0x63e, 0xc4203b00c0, 0xc42008c3c0, ...)
           	/go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:373 +0x1ac fp=0xc420663ce8 sp=0xc420663be8 pc=0xe1dbcc
    github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).publishEvents(0xc4201e6420, 0xc42054b080, 0x32, 0x63e, 0x0, 0x0, 0x0, 0x0, 0x0)
           	/go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:304 +0x14e fp=0xc420663e28 sp=0xc420663ce8 pc=0xe1d35e
    github.com/elastic/beats/libbeat/outputs/elasticsearch.(*Client).Publish(0xc4201e6420, 0x169f080, 0xc42040dc80, 0xc4200a4840, 0xc420663f18)
           	/go/src/github.com/elastic/beats/libbeat/outputs/elasticsearch/client.go:271 +0x43 fp=0xc420663e90 sp=0xc420663e28 pc=0xe1d173
    github.com/elastic/beats/libbeat/outputs.(*backoffClient).Publish(0xc420332f20, 0x169f080, 0xc42040dc80, 0x0, 0x0)
           	/go/src/github.com/elastic/beats/libbeat/outputs/backoff.go:60 +0x4b fp=0xc420663ed8 sp=0xc420663e90 pc=0xd6d3fb
    github.com/elastic/beats/libbeat/publisher/pipeline.(*netClientWorker).run(0xc420192780)
           	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:119 +0x324 fp=0xc420663fd8 sp=0xc420663ed8 pc=0xe8fe54
    runtime.goexit()
           	/usr/local/go/src/runtime/asm_amd64.s:2361 +0x1 fp=0xc420663fe0 sp=0xc420663fd8 pc=0x9e1891
    created by github.com/elastic/beats/libbeat/publisher/pipeline.makeClientWorker
           	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:48 +0xf0
```

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [February 14, 2019, 4:43pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/2 "2019-02-14T16:43:08Z")

</div>

This looks like a race condition due to some shared event fields being modified while being serialized.

Can you share your configs? Which processors are you using? did you configure `fields`?

---

<div class="post-metadata">

**Author:** ![mkorolyov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mkorolyov/32/40936_2.png) [@mkorolyov](https://discuss.elastic.co/u/mkorolyov)\
**Post date:** [February 15, 2019, 8:51am UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/3 "2019-02-15T08:51:06Z")

</div>

```
Config:
    filebeat.inputs:
    - prospector.type: log
      paths:
            - "/home/engine/log/app/stderr.log*"
      fields_under_root: true
      fields:
        host.ip_address: ${LOCAL_IP}
    - prospector.type: log
      paths:
            - "/home/engine/log/app/stdout.log*"
      close_removed: true
      close_renamed: true
      json.keys_under_root: true
      json.add_error_key: true
      fields_under_root: true
      fields:
        host.ip_address: ${LOCAL_IP}
    name: ${BEATNAME}
    setup.template.name: "filebeat-6.4.2"
    setup.template.pattern: "filebeat-6.4.2-*"
    setup.template.json.enabled: true
    setup.template.json.name: "filebeat-6.4.2"
    setup.template.json.path: "filebeat-6.4.2.template.json"
    setup.template.overwrite: false
    output.elasticsearch:
      hosts: ["http://${ELASTICSEARCH}:9300/"]
      index: "filebeat-%{[beat.version]}-%{+yyyy.MM.dd}"
```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [February 17, 2019, 4:58pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/4 "2019-02-17T16:58:16Z")

</div>

This may have been fixed in 6.4.3. From the [release notes](https://www.elastic.co/guide/en/beats/libbeat/6.6/release-notes-6.4.3.html):

> - Fix race condition when publishing monitoring data. [8646](https://github.com/elastic/beats/pull/8646)

---

<div class="post-metadata">

**Author:** ![mkorolyov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mkorolyov/32/40936_2.png) [@mkorolyov](https://discuss.elastic.co/u/mkorolyov)\
**Post date:** [February 18, 2019, 9:42am UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/5 "2019-02-18T09:42:17Z")

</div>

1. This fix was back ported to older versions.
2. As far as i understand after digging into stack traces and the code base it is not connected.

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [February 18, 2019, 7:16pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/6 "2019-02-18T19:16:08Z")

</div>

This is indeed a bug we haven't seen yet. It is caused by writing to the `host` namespace via the `fields` setting. The `host` namespace is builtin and uses some shared structure that is not always properly protected/copied it seems. The `fields.host` is also shared, and should not be modified. The race occurs when the builtin namespace modifies the `fields.host` provided structure.

Can you please open a bug report with version, configs and stack trace? [https://github.com/elastic/beats/issues/new/choose](https://github.com/elastic/beats/issues/new/choose)

There might be some (not so intuitive) workaround: Define a noop global process (e.g. drop a field that can't exist or define a when clause that can never be satisfied):

```auto

# define global processor with condition that is always false:
processors:
- drop_event.when.and:
    - has_fields: ["random"]
    - not.has_fields: ["random"]

```

This should force a copy of the "host" namespace per event, such that it can be safely merged with the builtin `host.name` field.

---

<div class="post-metadata">

**Author:** ![mkorolyov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mkorolyov/32/40936_2.png) [@mkorolyov](https://discuss.elastic.co/u/mkorolyov)\
**Post date:** [February 19, 2019, 2:58pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/7 "2019-02-19T14:58:45Z")

</div>

Thanks, created

> <https://github.com/elastic/beats/issues/10824>

---

<div class="post-metadata">

**Author:** ![steffens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/steffens/32/79630_2.png) [@steffens](https://discuss.elastic.co/u/steffens)\
**Post date:** [February 20, 2019, 3:37pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/8 "2019-02-20T15:37:23Z")

</div>

Thank you.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 20, 2019, 3:37pm UTC](https://discuss.elastic.co/t/panic-fatal-error-concurrent-map-iteration-and-map-write/168435/9 "2019-03-20T15:37:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.