Fillebeat keeps restarting in Kuberneates

Filebeat Version: 8.3.2 (image gets from docker.elastic.co/beats/filebeat:8.3.2)

I have roughly 40k logs in an NFS folder for Filebeat to collect. The Pod keeps restart itself in every ~10 minutes. And showing this error:

panic: reflect.Select: too many cases (max 65536)

goroutine 258 [running]:
reflect.Select({0xc0e5f7a000?, 0x10001?, 0x14124?})
	/usr/local/go/src/reflect/value.go:2793 +0x79a
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*Pipeline).runSignalPropagation(0xc0010827b8?)
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline.go:329 +0x1d8
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*Pipeline).registerSignalPropagation.func1
	/go/src/github.com/elastic/beats/libbeat/publisher/pipeline/pipeline.go:314 +0x96

I believe i have enough Memory, hundreds of GB. ulimit shows:

# ulimit -a
core file size          (blocks, -c) unlimited
data seg size           (kbytes, -d) unlimited
scheduling priority             (-e) 0
file size               (blocks, -f) unlimited
pending signals                 (-i) 3095242
max locked memory       (kbytes, -l) 64
max memory size         (kbytes, -m) unlimited
open files                      (-n) 1048576
pipe size            (512 bytes, -p) 8
POSIX message queues     (bytes, -q) 819200
real-time priority              (-r) 0
stack size              (kbytes, -s) 8192
cpu time               (seconds, -t) unlimited
max user processes              (-u) unlimited
virtual memory          (kbytes, -v) unlimited
file locks                      (-x) unlimited

Here is my filebeat.yaml inputs:

- type: filestream

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /mnt/*

  encoding: utf-8
  close_inactive: 1m
  tail_files: true
  file_identity.path: ~

Any hint? Thanks

FYI, this is similar to this case

anyone?

Hi,

There is an issue about reading from several thousands of files, it was closed, I have reopened it: https://github.com/elastic/beats/issues/16076

In the comments there is a trace that looks like yours.

Thanks @jsoriano. Yea, the log is similar.

One more clue, the same configuration worked fine when I was running Filebeat in a physical machine (RHEL 7.8). I am now suspending it is due to any misconfigure or limit in my Kuberneates setup.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.