Heartbeat fatal error: concurrent map iteration and map write

[Flavio_Fortes]

Hello,

I'm trying to run Heartbeat version 6.1.1 up to 6.1.3 all failing, normally after 20 minutes running due to the following error:

2018/03/19 23:56:00.640256 scheduler.go:170: DBG [scheduler] Next wakeup time: 2018-03-20 00:59:00.6347839 +0100 CET
fatal error: concurrent map iteration and map write

goroutine 14273 [running]:
runtime.throw(0x15f08b5, 0x26)
        /usr/local/go/src/runtime/panic.go:605 +0x95 fp=0xc4202daed8 sp=0xc4202daeb8 pc=0xa81f65
runtime.mapiternext(0xc4202db0d0)

Config:

# Configure monitors
heartbeat.monitors:

- type: tcp
  hosts: ["s1234", "s1234", "s1234", "s1234", "s1234", "s1234"]
  ports: [5555,9000]
  schedule: '@every 180s'
  timeout: 30s

- type: tcp
  hosts: ["s1234", "s12345"]
  ports: [12503]
  schedule: '@every 180s'
  timeout: 30s

#================================ General =====================================
# The name of the shipper that publishes the network data. It can be used to group
hostname: smonitor

#----------------------------- Logstash output --------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["localhost:5045"]

Running on
Platform : RHEL7 x86_64

Can someone help?
Kr,
Flavio

[andrewkroh]

This is definitely a bug. Is there any more to that stack trace than what you posted? If not, could you try running heartbeat in the foreground and add the GOTRACEBACK=all environment variable. Like

sudo GOTRACEBACK=all heartbeat -e

[Flavio_Fortes]

Thanks for the reply Andrew,

Attached the full stack trace running with " GOTRACEBACK=all heartbeat/heartbeat -e -d "*""

Kr,
Flavio

[andrewkroh]

Thanks that helps a lot it finding the root causes. Any chance you can provide it as text? You can use https://pastebin.com if it's too large to fit in here.

[andrewkroh]

I logged a bug in Github for this at https://github.com/elastic/beats/issues/6616#issue-307219993. I recommend subscribing to the issue for updates.