Fillebeat keeps restarting in Kuberneates

Elastic Stack Beats
1

Filebeat Version: 8.3.2 (image gets from docker.elastic.co/beats/filebeat:8.3.2)

I have roughly 40k logs in an NFS folder for Filebeat to collect. The Pod keeps restart itself in every ~10 minutes. And showing this error:

panic: reflect.Select: too many cases (max 65536)

goroutine 258 [running]:
reflect.Select({0xc0e5f7a000?, 0x10001?, 0x14124?})
	/usr/local/go/src/reflect/value.go:2793 +0x79a
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*Pipeline).runSignalPropagation(0xc0010827b8?)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline.go:329 +0x1d8
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*Pipeline).registerSignalPropagation.func1
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline.go:314 +0x96

I believe i have enough Memory, hundreds of GB. ulimit shows:

# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3095242
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Here is my filebeat.yaml inputs:

- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /mnt/*

  encoding: utf-8
  close_inactive: 1m
  tail_files: true
  file_identity.path: ~

Any hint? Thanks

FYI, this is similar to this case

2

anyone?

3

Hi,

There is an issue about reading from several thousands of files, it was closed, I have reopened it: https://github.com/elastic/beats/issues/16076

In the comments there is a trace that looks like yours.

4

Thanks @jsoriano. Yea, the log is similar.

One more clue, the same configuration worked fine when I was running Filebeat in a physical machine (RHEL 7.8). I am now suspending it is due to any misconfigure or limit in my Kuberneates setup.