Fatal error: runtime: cannot map pages in arena address space

Elastic Stack Beats
12

Currently looking at the log files. Some things I realised:

  • You are using Filebeat 1.2.2. I strongly recommend to update to 1.2.3
  • Can you share your config file? It seems like you have quite a few prospectors.

@andrewkroh @steffens There are some interesting things in the log files. The first few messages here:

runtime: failed to create new OS thread (have 3055 already; errno=8)
fatal error: runtime.newosproc
runtime: failed to create new OS thread (have 3055 already; errno=8)
fatal error: runtime.newosproc

runtime stack:
runtime.throw(0x8452f0, 0x11)
	/usr/local/go/src/runtime/panic.go:527 +0x7f
runtime.newosproc(0x1cb87080, 0x0)
	/usr/local/go/src/runtime/os1_windows.go:278 +0x128
runtime.newm(0x0, 0x12760a00)
	/usr/local/go/src/runtime/proc1.go:1105 +0xf0
runtime.startm(0x12760a00, 0x0)
	/usr/local/go/src/runtime/proc1.go:1178 +0xd1
runtime.handoffp(0x12760a00)
	/usr/local/go/src/runtime/proc1.go:1201 +0x27c
runtime.retake(0x50595fa0, 0x118dd0, 0x0)
	/usr/local/go/src/runtime/proc1.go:3104 +0x1e4
runtime.sysmon()
	/usr/local/go/src/runtime/proc1.go:3031 +0x25c
runtime.mstart1()
	/usr/local/go/src/runtime/proc1.go:721 +0xc9
runtime.mstart()
	/usr/local/go/src/runtime/proc1.go:691 +0x59

goroutine 1 [runnable]:
github.com/elastic/beats/filebeat/crawler.(*Crawler).Start(0x192ede68, 0x1296c000, 0x15, 0x15, 0x127566c0)
	/go/src/github.com/elastic/beats/filebeat/crawler/crawler.go:60 +0x4df
github.com/elastic/beats/filebeat/beat.(*Filebeat).Run(0x12759b70, 0x127528d0, 0x0, 0x0)
	/go/src/github.com/elastic/beats/filebeat/beat/filebeat.go:92 +0x3ad
github.com/elastic/beats/libbeat/beat.(*Beat).Run(0x127528d0)
	/go/src/github.com/elastic/beats/libbeat/beat/beat.go:176 +0x250
github.com/elastic/beats/libbeat/beat.Run(0x82a430, 0x8, 0x802fb8, 0x5, 0x327648d8, 0x12759b70, 0x1276e700)
	/go/src/github.com/elastic/beats/libbeat/beat/beat.go:86 +0x1c1
main.main()
	/go/src/github.com/elastic/beats/filebeat/main.go:21 +0x86

goroutine 5 [syscall]:
os/signal.loop()
	/usr/local/go/src/os/signal/signal_unix.go:22 +0x1a
created by os/signal.init.1
	/usr/local/go/src/os/signal/signal_unix.go:28 +0x36

@ori.rubinfeld You posted this issue also here with the above messages: Filebeat crashes on windows servers Are these 2 threads for the same issue?

It seems that everything runs fine until the interesting limit of 3054 threads is hit.