Elastic Agent v8.7.0, Filebeat UDP listener error

Elastic Stack Elastic Agent
1

Hi
After upgrading from 8.6.2 to 8.7.0 this error happens all the time causing the agent/filebeat to crash and restart:

nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.090+0200","message":"panic: runtime error: index out of range [1] with length 0","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}
nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.090+0200","message":"goroutine 160 [running]:","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}
nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.091+0200","message":"github.com/elastic/beats/v7/filebeat/input/udp.procNetUDP({0xc00088d290?, 0x1, 0x1})","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}
nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.091+0200","message":"github.com/elastic/beats/v7/filebeat/input/udp/input.go:271 +0x552","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}
nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.091+0200","message":"github.com/elastic/beats/v7/filebeat/input/udp.(*inputMetrics).poll(0xc000762e80, {0xc00088d290, 0x1, 0x1}, 0x2020202020202020?, 0xc00088d020)","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}
nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.091+0200","message":"github.com/elastic/beats/v7/filebeat/input/udp/input.go:242 +0xd1","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}
nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.091+0200","message":"created by github.com/elastic/beats/v7/filebeat/input/udp.newInputMetrics","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}
nagent  | {"log.level":"error","@timestamp":"2023-04-07T12:57:29.091+0200","message":"github.com/elastic/beats/v7/filebeat/input/udp/input.go:216 +0xa70","component":{"binary":"filebeat","dataset":"elastic_agent.filebeat","id":"udp-default","type":"udp"},"log":{"source":"udp-default"},"ecs.version":"1.6.0"}

The only UDP listener I have is from the pfSense integration which has always worked flawlessly, inlcuding at version 8.6.2. If I rollback the agent to 8.6.2 it all works well again.

I researched breaking changes and whatsoever but didn't find anything useful pointing in a new configuration direction or anything I may be missing. What could be the reason? Is this a version bug?

2

Hi,
Same issue here. Clean configuration, only fortinet module enabled. Tried both udp and tcp same result. Rolled back and everything started working again.

panic: runtime error: index out of range [1] with length 0
goroutine 434 [running]:
github.com/elastic/beats/v7/filebeat/input/tcp.procNetTCP({0xc0009b4450?, 0x1, 0x1})
github.com/elastic/beats/v7/filebeat/input/tcp/input.go:277 +0x47b
github.com/elastic/beats/v7/filebeat/input/tcp.(*inputMetrics).poll(0xc000e67490, {0xc0009b4450, 0x1, 0x1}, 0x55dfb5a0bcc6?, 0xc0009b4240)
github.com/elastic/beats/v7/filebeat/input/tcp/input.go:249 +0xd1
created by github.com/elastic/beats/v7/filebeat/input/tcp.newInputMetrics
github.com/elastic/beats/v7/filebeat/input/tcp/input.go:223 +0x9d0

3

I guess we can establish that it's a filebeat tcp/udp listener bug then. Hope someone picks this up. Don't understand how it slipped (unit) tests in the first place to be honest?

4

I would suggest that you open a bug report in the github repository for the Elastic Agent providing the steps to replicate.

But it looks like a bug on the Kibana side as the code for the inputs has not changed in the last 3 months.

5

How come that it's still only happening on the newest agent version and not an older one? Kibana remains the newest 8.7.0 for both configurations. Do you advise opening a Kibana ticket still, rather than an agent one?

6

Well, maybe it is in issue in the Agent in the end, but unrelated to the input code for tcp and udp inputs, that it what has not changed according to github commits.

Since it is happening on an integration, you may open an bug report on the Elastic Agent repository and people from elastic you change it if it is not an issue with the Agent.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.