Filebeat with Large Registry/State File Causes `runtime: program exceeds 10000-thread limit` on Startup

Hi All,

I'm seeing an issue, where a Filebeat instead with a large registrar (~33k entries).

{"level":"info","timestamp":"2022-01-29T00:32:41.799Z","logger":"registrar","caller":"registrar/registrar.go:109","message":"States Loaded from registrar: 33156"

When it restarts will eventually crash with the following error:

(I've truncated the log as it is extremely large; ~20k lines, I can provide full log if needed)

runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion
runtime stack:
runtime.throw({0x55f408026f08, 0x0})
/usr/local/go/src/runtime/panic.go:1198 +0x71
runtime.checkmcount()
/usr/local/go/src/runtime/proc.go:760 +0x8c
runtime.mReserveID()
/usr/local/go/src/runtime/proc.go:776 +0x36
runtime.startm(0xc0000e4800, 0x0)
/usr/local/go/src/runtime/proc.go:2477 +0x90
runtime.handoffp(0xffffffff)
/usr/local/go/src/runtime/proc.go:2519 +0x2ac
runtime.retake(0xa21a58dc493)
/usr/local/go/src/runtime/proc.go:5537 +0x1e7
runtime.sysmon()
/usr/local/go/src/runtime/proc.go:5445 +0x337
runtime.mstart1()
/usr/local/go/src/runtime/proc.go:1407 +0x95
runtime.mstart0()
/usr/local/go/src/runtime/proc.go:1365 +0x79
runtime.mstart()
/usr/local/go/src/runtime/asm_amd64.s:248 +0x5
goroutine 1 [chan receive, 2 minutes]:
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Wait(...)
/go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:45
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run(0xc001028cc0, 0xc000ce6000)
/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:512 +0x1718
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc000ce6000, {{0x55f40800dca7, 0x8}, {0x55f40800dca7, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, ...}, ...}, ...}, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:498 +0x90b
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0xc000c4dd28, 0x8)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:215 +0x628
github.com/elastic/beats/v7/libbeat/cmd/instance.Run({{0x55f40800dca7, 0x8}, {0x55f40800dca7, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:216 +0xda
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc000806000, {0x55f40800614b, 0x3, 0x3})
/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x58
github.com/spf13/cobra.(*Command).execute(0xc000806000, {0xc000050090, 0x3, 0x3})
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc000806000)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
/go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22 +0x25

I've also noticed that it consumes an insane amount of RAM on startup:

Filebeat.yml

http:
  enabled: true
  port: 5066
  host: 0.0.0.0
filebeat:
  inputs:
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/prod/cdr*
    tags: ["cucm","prod","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/prod/cmr*
    tags: ["cucm","prod","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/hi/cdr*
    tags: ["cucm","hi","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/hi/cmr*
    tags: ["cucm","hi","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/dev/cdr*
    tags: ["cucm","dev","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/dev/cdr*
    tags: ["cucm","dev","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
logging.json: true
output:
  logstash:
    enabled: ${LOGSTASH_ENABLED:false}
    hosts: "${LOGSTASH_HOSTS:\
  [<snipped>]}"
    loadbalance: ${LOGSTASH_LB:true}
monitoring:
  enabled: false
  cluster_uuid: ${MONITORING_CLUSTER_UUID}

I managed to get it stabilized, by increasing CPU from 2 to 4 and RAM from 4GB to 8GB, but I still think there is an issue here, as this amount of resource seems pretty excessive to get Filebeat to startup.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.