# Filebeat with Large Registry/State File Causes \`runtime: program exceeds 10000-thread limit\` on Startup

**URL:** https://discuss.elastic.co/t/filebeat-with-large-registry-state-file-causes-runtime-program-exceeds-10000-thread-limit-on-startup/295738
**Category:** Beats
**Tags:** filebeat
**Created:** [January 29, 2022, 12:49am UTC](https://discuss.elastic.co/t/filebeat-with-large-registry-state-file-causes-runtime-program-exceeds-10000-thread-limit-on-startup/295738 "2022-01-29T00:49:35Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [January 29, 2022, 12:49am UTC](https://discuss.elastic.co/t/filebeat-with-large-registry-state-file-causes-runtime-program-exceeds-10000-thread-limit-on-startup/295738/1 "2022-01-29T00:49:35Z")

</div>

Hi All,

I'm seeing an issue, where a Filebeat instead with a large registrar (~33k entries).

```auto
{"level":"info","timestamp":"2022-01-29T00:32:41.799Z","logger":"registrar","caller":"registrar/registrar.go:109","message":"States Loaded from registrar: 33156"

```

When it restarts will eventually crash with the following error:

(I've truncated the log as it is extremely large; ~20k lines, I can provide full log if needed)

```auto
runtime: program exceeds 10000-thread limit
fatal error: thread exhaustion
runtime stack:
runtime.throw({0x55f408026f08, 0x0})
/usr/local/go/src/runtime/panic.go:1198 +0x71
runtime.checkmcount()
/usr/local/go/src/runtime/proc.go:760 +0x8c
runtime.mReserveID()
/usr/local/go/src/runtime/proc.go:776 +0x36
runtime.startm(0xc0000e4800, 0x0)
/usr/local/go/src/runtime/proc.go:2477 +0x90
runtime.handoffp(0xffffffff)
/usr/local/go/src/runtime/proc.go:2519 +0x2ac
runtime.retake(0xa21a58dc493)
/usr/local/go/src/runtime/proc.go:5537 +0x1e7
runtime.sysmon()
/usr/local/go/src/runtime/proc.go:5445 +0x337
runtime.mstart1()
/usr/local/go/src/runtime/proc.go:1407 +0x95
runtime.mstart0()
/usr/local/go/src/runtime/proc.go:1365 +0x79
runtime.mstart()
/usr/local/go/src/runtime/asm_amd64.s:248 +0x5
goroutine 1 [chan receive, 2 minutes]:
github.com/elastic/beats/v7/filebeat/beater.(*signalWait).Wait(...)
/go/src/github.com/elastic/beats/filebeat/beater/signalwait.go:45
github.com/elastic/beats/v7/filebeat/beater.(*Filebeat).Run(0xc001028cc0, 0xc000ce6000)
/go/src/github.com/elastic/beats/filebeat/beater/filebeat.go:512 +0x1718
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc000ce6000, {{0x55f40800dca7, 0x8}, {0x55f40800dca7, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, ...}, ...}, ...}, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:498 +0x90b
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0xc000c4dd28, 0x8)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:215 +0x628
github.com/elastic/beats/v7/libbeat/cmd/instance.Run({{0x55f40800dca7, 0x8}, {0x55f40800dca7, 0x8}, {0x0, 0x0}, 0x1, 0x1, {{0x0, 0x0}, ...}, ...}, ...)
/go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:216 +0xda
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc000806000, {0x55f40800614b, 0x3, 0x3})
/go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x58
github.com/spf13/cobra.(*Command).execute(0xc000806000, {0xc000050090, 0x3, 0x3})
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:846 +0x5f8
github.com/spf13/cobra.(*Command).ExecuteC(0xc000806000)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:950 +0x3ad
github.com/spf13/cobra.(*Command).Execute(...)
/go/pkg/mod/github.com/spf13/cobra@v1.0.0/command.go:887
main.main()
/go/src/github.com/elastic/beats/x-pack/filebeat/main.go:22 +0x25

```

I've also noticed that it consumes an insane amount of RAM on startup:

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/2/3/2358f42e63dfb3ceb98c1b5cde01a974224ebffb.png)

Filebeat.yml

```auto
http:
  enabled: true
  port: 5066
  host: 0.0.0.0
filebeat:
  inputs:
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/prod/cdr*
    tags: ["cucm","prod","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/prod/cmr*
    tags: ["cucm","prod","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/hi/cdr*
    tags: ["cucm","hi","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/hi/cmr*
    tags: ["cucm","hi","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/dev/cdr*
    tags: ["cucm","dev","cdr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
  - type: filestream
    enabled: true
    paths:
      - /home/cucm/uploads/dev/cdr*
    tags: ["cucm","dev","cmr"]
    close.on_state_change.inactive: 5m
    close.on_state_change.removed: true
    clean_removed: true
    ignore_older: 168h
    clean_inactive: 169h
logging.json: true
output:
  logstash:
    enabled: ${LOGSTASH_ENABLED:false}
    hosts: "${LOGSTASH_HOSTS:\
  [<snipped>]}"
    loadbalance: ${LOGSTASH_LB:true}
monitoring:
  enabled: false
  cluster_uuid: ${MONITORING_CLUSTER_UUID}

```

---

<div class="post-metadata">

### Author: ![BenB196](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/benb196/32/83401_2.png) [@BenB196](https://discuss.elastic.co/u/BenB196)
#### Post date: [January 29, 2022, 1:29am UTC](https://discuss.elastic.co/t/filebeat-with-large-registry-state-file-causes-runtime-program-exceeds-10000-thread-limit-on-startup/295738/2 "2022-01-29T01:29:55Z")

</div>

I managed to get it stabilized, by increasing CPU from `2` to `4` and RAM from `4GB` to `8GB`, but I still think there is an issue here, as this amount of resource seems pretty excessive to get Filebeat to startup.

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/a/c/acd6788a45fff6f7a65752a594ba5ca90b0b6de5.png)

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [February 26, 2022, 3:30am UTC](https://discuss.elastic.co/t/filebeat-with-large-registry-state-file-causes-runtime-program-exceeds-10000-thread-limit-on-startup/295738/3 "2022-02-26T03:30:13Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.