Fillebeat keeps restarting in Kuberneates

[Jehutywong]

Filebeat Version: 8.3.2 (image gets from docker.elastic.co/beats/filebeat:8.3.2)

I have roughly 40k logs in an NFS folder for Filebeat to collect. The Pod keeps restart itself in every ~10 minutes. And showing this error:

panic: reflect.Select: too many cases (max 65536)

goroutine 258 [running]:
reflect.Select({0xc0e5f7a000?, 0x10001?, 0x14124?})
	/usr/local/go/src/reflect/value.go:2793 +0x79a
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*Pipeline).runSignalPropagation(0xc0010827b8?)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline.go:329 +0x1d8
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*Pipeline).registerSignalPropagation.func1
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline.go:314 +0x96

I believe i have enough Memory, hundreds of GB. ulimit shows:

# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3095242
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Here is my filebeat.yaml inputs:

- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /mnt/*

  encoding: utf-8
  close_inactive: 1m
  tail_files: true
  file_identity.path: ~

Any hint? Thanks

FYI, this is similar to this case

Filebeat is leaking memory

There are about 100,000 log files on the VM, the characteristics of the VM OS Windows Server 2019, CPU 4, RAM 4GB. To collect logs and then send them to logstash, filebeat 7.16.0 is used. After starting filebeat, it begins to consume RAM memory to the maximum, as a result, the filebeat service stops, collecting logs ends with an error: ERROR [logstash] logstash/async.go:280 Failed to publish events caused by: write tcp 10.0.10.5:53536->10.0.10.6:5044: wsasend: An existing connection was forcibl…

[Jehutywong]

anyone?

[jsoriano (Jaime Soriano)]

Hi,

There is an issue about reading from several thousands of files, it was closed, I have reopened it: https://github.com/elastic/beats/issues/16076

In the comments there is a trace that looks like yours.

[Jehutywong]

Thanks @jsoriano. Yea, the log is similar.

One more clue, the same configuration worked fine when I was running Filebeat in a physical machine (RHEL 7.8). I am now suspending it is due to any misconfigure or limit in my Kuberneates setup.