Filebeat with Large Registry/State File Causes `runtime: program exceeds 10000-thread limit` on Startup

[BenB196]

Hi All,

I'm seeing an issue, where a Filebeat instead with a large registrar (~33k entries).

{"level":"info","timestamp":"2022-01-29T00:32:41.799Z","logger":"registrar","caller":"registrar/registrar.go:109","message":"States Loaded from registrar: 33156"

When it restarts will eventually crash with the following error:

(I've truncated the log as it is extremely large; ~20k lines, I can provide full log if needed)

runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion
runtime stack:
runtime.throw({0x55f408026f08, 0x0})
/usr/local/go/src/runtime/panic.go:1198 +0x71
runtime.checkmcount()
/usr/local/go/src/runtime/proc.go:760 +0x8c
runtime.mReserveID()
/usr/local/go/src/runtime/proc.go:776 +0x36
runtime.startm(0xc0000e4800, 0x0)
/usr/local/go/src/runtime/proc.go:2477 +0x90
runtime.handoffp(0xffffffff)
/usr/local/go/src/runtime/proc.go:2519 +0x2ac
runtime.retake(0xa21a58dc493)
/usr/local/go/src/runtime/proc.go:5537 +0x1e7
runtime.sysmon()
/usr/local/go/src/runtime/proc.go:5445 +0x337
runtime.mstart1()
/usr/local/go/src/runtime/proc.go:1407 +0x95
runtime.mstart0()
/usr/local/go/src/runtime/proc.go:1365 +0x79
runtime.mstart()
/usr/local/go/src/runtime/asm_amd64.s:248 +0x5
goroutine 1 [chan receive, 2 minutes]:
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Wait(...)
/go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:45
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run(0xc001028cc0, 0xc000ce6000)
/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:512 +0x1718
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc000ce6000, {{0x55f40800dca7, 0x8}, {0x55f40800dca7, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, ...}, ...}, ...}, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:498 +0x90b
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0xc000c4dd28, 0x8)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:215 +0x628
github.com/elastic/beats/v7/libbeat/cmd/instance.Run({{0x55f40800dca7, 0x8}, {0x55f40800dca7, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:216 +0xda
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc000806000, {0x55f40800614b, 0x3, 0x3})
/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x58
github.com/spf13/cobra.(*Command).execute(0xc000806000, {0xc000050090, 0x3, 0x3})
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc000806000)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
/go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22 +0x25

I've also noticed that it consumes an insane amount of RAM on startup:

Filebeat.yml

http:
  enabled: true
  port: 5066
  host: 0.0.0.0
filebeat:
  inputs:
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/prod/cdr*
    tags: ["cucm","prod","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/prod/cmr*
    tags: ["cucm","prod","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/hi/cdr*
    tags: ["cucm","hi","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/hi/cmr*
    tags: ["cucm","hi","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/dev/cdr*
    tags: ["cucm","dev","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/dev/cdr*
    tags: ["cucm","dev","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
logging.json: true
output:
  logstash:
    enabled: ${LOGSTASH_ENABLED:false}
    hosts: "${LOGSTASH_HOSTS:\
  [<snipped>]}"
    loadbalance: ${LOGSTASH_LB:true}
monitoring:
  enabled: false
  cluster_uuid: ${MONITORING_CLUSTER_UUID}

[BenB196]

I managed to get it stabilized, by increasing CPU from 2 to 4 and RAM from 4GB to 8GB, but I still think there is an issue here, as this amount of resource seems pretty excessive to get Filebeat to startup.