Heartbeat keeps failing after upgrade from 7.4.2 to 7.5.2

Elastic Stack Beats
1

Heartbeat keeps failing after upgrade from 7.4.2 to 7.5.2.
This is the stack trace of it.

2020-01-26T15:40:45.261Z	INFO	[publisher]	pipeline/module.go:97	Beat name: beat001-fr
2020-01-26T15:40:45.262Z	INFO	instance/beat.go:429	heartbeat start running.
2020-01-26T15:40:45.262Z	INFO	beater/heartbeat.go:80	heartbeat is running! Hit CTRL-C to stop it.
2020-01-26T15:40:45.262Z	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-01-26T15:40:45.262Z	INFO	cfgfile/reload.go:171	Config reloader started
2020-01-26T15:40:55.302Z	INFO	[reload]	cfgfile/list.go:118	Stopping 35 runners ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1438bf4]

goroutine 40 [running]:
github.com/elastic/beats/heartbeat/monitors/active/http.(*Config).Validate.func1(0xc00021a9c0, 0x11, 0x0, 0x2bfcd20)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/config.go:172 +0x34
github.com/elastic/beats/heartbeat/monitors/active/http.(*Config).Validate(0xc0005d45a0, 0x25efe20, 0xc0005d45a0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/config.go:185 +0x26a
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.tryValidate(0x27e2680, 0xc0005d45a0, 0x199, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/validator.go:123 +0xf4
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyStruct(0xc000488480, 0x27e2680, 0xc0005d4480, 0x199, 0xc00061a3c0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:288 +0xa42
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyInto(0xc000488480, 0x25efe20, 0xc0005d4480, 0x16, 0xc00061a3c0, 0x120, 0x27e2680)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:161 +0x3fd
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*Config).Unpack(0xc00061a3c0, 0x25efe20, 0xc0005d4480, 0xc00000e020, 0x4, 0x4, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:144 +0x104
github.com/elastic/beats/libbeat/common.(*Config).Unpack(...)
	/go/src/github.com/elastic/beats/libbeat/common/config.go:180
github.com/elastic/beats/heartbeat/monitors/active/http.create(0x2889f0c, 0x4, 0xc00061a3c0, 0x2, 0x2, 0xc000766a40, 0x1c, 0x1, 0x0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/http.go:46 +0xcc
github.com/elastic/beats/heartbeat/monitors.(*pluginBuilder).create(...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/plugin.go:157
github.com/elastic/beats/heartbeat/monitors.newMonitorUnsafe(0xc00061a3c0, 0xc000010130, 0x7f49ae331840, 0xc0000d42c0, 0xc00017a100, 0xc00061a300, 0x0, 0xc0001e3af0, 0xfa5b05, 0x2643740)
	/go/src/github.com/elastic/beats/heartbeat/monitors/monitor.go:165 +0x3ae
github.com/elastic/beats/heartbeat/monitors.newMonitor(0xc00061a3c0, 0xc000010130, 0x7f49ae331840, 0xc0000d42c0, 0xc00017a100, 0x0, 0x0, 0x0, 0x0, 0xc0000d0ba0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/monitor.go:106 +0x7c
github.com/elastic/beats/heartbeat/monitors.(*RunnerFactory).Create(0xc0001f69a0, 0x2c30ec0, 0xc0000d42c0, 0xc00061a3c0, 0x0, 0x1, 0x1, 0x0, 0x0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/factory.go:41 +0x95
github.com/elastic/beats/libbeat/cfgfile.(*RunnerList).Reload(0xc00026d220, 0xc00042a000, 0x61, 0x80, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cfgfile/list.go:94 +0x852
github.com/elastic/beats/libbeat/cfgfile.(*Reloader).Run(0xc00017cea0, 0x2bf9e40, 0xc0001f69a0)
	/go/src/github.com/elastic/beats/libbeat/cfgfile/reload.go:218 +0x3cb
created by github.com/elastic/beats/heartbeat/beater.(*Heartbeat).RunReloadableMonitors
	/go/src/github.com/elastic/beats/heartbeat/beater/heartbeat.go:150 +0x92
2

I just migrated back to 7.4.2 for the heartbeats. Works again. Any ideas what could be wrong?
Rest of the stack went flawless.

3

I setup a clean server and heartbeat 7.5.2. I am getting this error. Are there more people finding this issue? I am reloading the configs every 10s (basic settings)

2020-01-27T11:18:44.596Z	INFO	[reload]	cfgfile/list.go:118	Stopping 66 runners ...
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1438bf4]

goroutine 47 [running]:
github.com/elastic/beats/heartbeat/monitors/active/http.(*Config).Validate.func1(0xc000520ce0, 0x11, 0x0, 0x2bfcd20)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/config.go:172 +0x34
github.com/elastic/beats/heartbeat/monitors/active/http.(*Config).Validate(0xc00063c480, 0x25efe20, 0xc00063c480)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/config.go:185 +0x26a
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.tryValidate(0x27e2680, 0xc00063c480, 0x199, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/validator.go:123 +0xf4
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyStruct(0xc000554980, 0x27e2680, 0xc00063c360, 0x199, 0xc00061dd70, 0x0, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:288 +0xa42
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.reifyInto(0xc000554980, 0x25efe20, 0xc00063c360, 0x16, 0xc00061dd70, 0x120, 0x27e2680)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:161 +0x3fd
github.com/elastic/beats/vendor/github.com/elastic/go-ucfg.(*Config).Unpack(0xc00061dd70, 0x25efe20, 0xc00063c360, 0xc00000ef00, 0x4, 0x4, 0x1, 0x0)
	/go/src/github.com/elastic/beats/vendor/github.com/elastic/go-ucfg/reify.go:144 +0x104
github.com/elastic/beats/libbeat/common.(*Config).Unpack(...)
	/go/src/github.com/elastic/beats/libbeat/common/config.go:180
github.com/elastic/beats/heartbeat/monitors/active/http.create(0x2889f0c, 0x4, 0xc00061dd70, 0x2, 0x2, 0xc0003aab60, 0x1c, 0x1, 0x0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/active/http/http.go:46 +0xcc
github.com/elastic/beats/heartbeat/monitors.(*pluginBuilder).create(...)
	/go/src/github.com/elastic/beats/heartbeat/monitors/plugin.go:157
github.com/elastic/beats/heartbeat/monitors.newMonitorUnsafe(0xc00061dd70, 0xc000010088, 0x7f97810379a0, 0xc0005538c0, 0xc0002b8880, 0xc00061dd00, 0x0, 0xc0006e5af0, 0xfa5b05, 0x2643740)
	/go/src/github.com/elastic/beats/heartbeat/monitors/monitor.go:165 +0x3ae
github.com/elastic/beats/heartbeat/monitors.newMonitor(0xc00061dd70, 0xc000010088, 0x7f97810379a0, 0xc0005538c0, 0xc0002b8880, 0x0, 0x0, 0x0, 0x0, 0xc0000f2cc0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/monitor.go:106 +0x7c
github.com/elastic/beats/heartbeat/monitors.(*RunnerFactory).Create(0xc0003834a0, 0x2c30ec0, 0xc0005538c0, 0xc00061dd70, 0x0, 0x1, 0x1, 0x0, 0x0)
	/go/src/github.com/elastic/beats/heartbeat/monitors/factory.go:41 +0x95
github.com/elastic/beats/libbeat/cfgfile.(*RunnerList).Reload(0xc0000e0050, 0xc0004e8800, 0x63, 0x80, 0x0, 0x0)
	/go/src/github.com/elastic/beats/libbeat/cfgfile/list.go:94 +0x852
github.com/elastic/beats/libbeat/cfgfile.(*Reloader).Run(0xc0001de8a0, 0x2bf9e40, 0xc0003834a0)
	/go/src/github.com/elastic/beats/libbeat/cfgfile/reload.go:218 +0x3cb
created by github.com/elastic/beats/heartbeat/beater.(*Heartbeat).RunReloadableMonitors
	/go/src/github.com/elastic/beats/heartbeat/beater/heartbeat.go:150 +0x92
4

After some debugging I found out that when a monitor has a hosts location which contains a space or no correct url format. All the monitors are not active and makes the reloader crash.

The older version send it is a down status, with error message of name resolving.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.