Packetbeat: runtime error: invalid memory address or nil pointer dereference after running for ~16 hours

[salekseev (Stas Alekseev)]

• Version: 5.4.2
• Operating System: 3.8.13-118.14.1.el6uek.x86_64

2017-06-26T20:31:01-04:00 ERR ParseMemcache(UDP) exception. Recovering, but please report this: runtime error: invalid memory address or nil pointer dereference.
2017-06-26T20:31:01-04:00 ERR Stacktrace: goroutine 36 [running]:
runtime/debug.Stack(0xbcffd8, 0x2b, 0xc42128d6f8)
/usr/local/go/src/runtime/debug/stack.go:24 +0x79
github.com/elastic/beats/libbeat/logp.Recover(0xbc4448, 0x1c)
/go/src/github.com/elastic/beats/libbeat/logp/log.go:145 +0x138
panic(0xad4800, 0xc420014050)
/usr/local/go/src/runtime/panic.go:458 +0x243
github.com/elastic/beats/packetbeat/protos/memcache.(*memcache).onUDPMessage(0xc42127a780, 0xc4211a5c00, 0xc422116858, 0x11a5c01, 0x0, 0x0, 0x0, 0x0)
/go/src/github.com/elastic/beats/packetbeat/protos/memcache/plugin_udp.go:163 +0x61
github.com/elastic/beats/packetbeat/protos/memcache.(*memcache).ParseUDP(0xc42127a780, 0xc422116840)
/go/src/github.com/elastic/beats/packetbeat/protos/memcache/plugin_udp.go:122 +0x80d
github.com/elastic/beats/packetbeat/protos/udp.(*UDP).Process(0xc421256f20, 0x0, 0xc422116840)
/go/src/github.com/elastic/beats/packetbeat/protos/udp/udp.go:59 +0x293
github.com/elastic/beats/packetbeat/decoder.(*Decoder).onUDP(0xc421292000, 0xc422116840)
/go/src/github.com/elastic/beats/packetbeat/decoder/decoder.go:295 +0xd9
github.com/elastic/beats/packetbeat/decoder.(*Decoder).process(0xc421292000, 0xc422116840, 0x2d, 0x6c, 0x1177680, 0xc421292000)
/go/src/github.com/elastic/beats/packetbeat/decoder/decoder.go:253 +0x299
github.com/elastic/beats/packetbeat/decoder.(*Decoder).OnPacket(0xc421292000, 0xc42271788c, 0x6c, 0x6c, 0xc4217555c0)
/go/src/github.com/elastic/beats/packetbeat/decoder/decoder.go:164 +0x343
github.com/elastic/beats/packetbeat/sniffer.(*SnifferSetup).Run(0xc421286360, 0xc20018, 0xc42127fca0)
/go/src/github.com/elastic/beats/packetbeat/sniffer/sniffer.go:381 +0x493
github.com/elastic/beats/packetbeat/beater.(*packetbeat).Run.func2(0xc42127fca0, 0xc420079540, 0xc421286600)
/go/src/github.com/elastic/beats/packetbeat/beater/packetbeat.go:155 +0x6a
created by github.com/elastic/beats/packetbeat/beater.(*packetbeat).Run
/go/src/github.com/elastic/beats/packetbeat/beater/packetbeat.go:159 +0x1b2

After which the process dies.

[steffens (Steffen Siering)]

The error message about the panic is generated by packetbeat. If this panic occurs, packetbeat recovers processing by dropping the state of the stream affected and continue processing more packets.

When packetbeat dies due to a panic it can not recover from, packetbeat will not be able to write to the log file. But the go-runtime will print a stack trace to stderr. Do you log to stdout/stderr or to file? Have you also check the kernel logs for a reason packetbeat has been stopped? Does it happen often?

Which sniffer type are you using. Normally errors happen if the analyzer is not in sync with the actual network stream and parser/processing is not strict enough on invalid values. Often times these kind of issues can be triggered on startup or after packet loss. The af_packet sniffer type performs somewhat better then the default pcap based, reducing the chance of packet loss.

About the 'Recovering' message, some sample traffic for reproducing would be helpful.