Metricbeat service get stopped after service start

Metricbeat version: 7.9.3

Below is error, when i run the metricbeat command in the server.

runtime/cgo: pthread_create failed: Operation not permitted
SIGABRT: abort
PC=0x7f2f78896a7c m=2 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f2f78896a7c
stack: frame={sp:0x7f2f51ced7e0, fp:0x0} stack=[0x7f2f514ee1e8,0x7f2f51cedde8)
00007f2f51ced6e0:  0000000001e06550 <runtime.netpoll+464>  00007f2f51ced718 
00007f2f51ced6f0:  00007f2f504b4c80  0000000000000077 
00007f2f51ced700:  0000000000000004  0000000400000000 
00007f2f51ced710:  0000000000000003  000000c000173e00 
00007f2f51ced720:  0000000000000000  0000000000000000 
00007f2f51ced730:  504b4f2000000004  0000000400007f2f 
00007f2f51ced740:  00007f2f504b4e40  504b4d6000000004 
00007f2f51ced750:  0000000400007f2f  00007f2f504b4c80 
00007f2f51ced760:  0000000000000000  0000000000000000 
00007f2f51ced770:  0000000000000000  0000000000000130 
00007f2f51ced780:  0000000000000000  0000000000000000 
00007f2f51ced790:  0000001300000000  0000000000000120 
00007f2f51ced7a0:  0000000000000000  0000000000000000 
00007f2f51ced7b0:  0000000000000000  0000000000000000 
00007f2f51ced7c0:  0000000000000004  0000003400000013 
00007f2f51ced7d0:  0000000000000000  00007f2f78896a6e 
00007f2f51ced7e0: <0000000000000000  000000770000007c 
00007f2f51ced7f0:  0000005b0000006e  0000000000000000 
00007f2f51ced800:  0000000000000000  00007f2f78926a51 
00007f2f51ced810:  00007f2f437fe640  00007f2f51cedaf0 
00007f2f51ced820:  00007f2f51ced96e  00007f2f51ced96f 
00007f2f51ced830:  0000000000000000  00007f2f78894759 
00007f2f51ced840:  00000000007fff00  0000000000000000 
00007f2f51ced850:  00000000003d0f00  00007f2f437fe910 
00007f2f51ced860:  00007f2f437fe910  951972223d314e00 
00007f2f51ced870:  00007f2f51cee640  0000000000000006 
00007f2f51ced880:  0000000009ca52e0  0000000000000000 
00007f2f51ced890:  00000000058d3c90  00007f2f78842476 
00007f2f51ced8a0:  00007f2f78a1ae90  00007f2f788287f3 
00007f2f51ced8b0:  0000000000000020  00007f2f437fe640 
00007f2f51ced8c0:  0000000000000000  0000000000000001 
00007f2f51ced8d0:  00007f2f437fe640  00007f2f788955c4 
runtime: unknown pc 0x7f2f78896a7c
stack: frame={sp:0x7f2f51ced7e0, fp:0x0} stack=[0x7f2f514ee1e8,0x7f2f51cedde8)
00007f2f51ced6e0:  0000000001e06550 <runtime.netpoll+464>  00007f2f51ced718 
00007f2f51ced6f0:  00007f2f504b4c80  0000000000000077 
00007f2f51ced700:  0000000000000004  0000000400000000 
00007f2f51ced710:  0000000000000003  000000c000173e00 
00007f2f51ced720:  0000000000000000  0000000000000000 
00007f2f51ced730:  504b4f2000000004  0000000400007f2f 
00007f2f51ced740:  00007f2f504b4e40  504b4d6000000004 
00007f2f51ced750:  0000000400007f2f  00007f2f504b4c80 
00007f2f51ced760:  0000000000000000  0000000000000000 
00007f2f51ced770:  0000000000000000  0000000000000130 
00007f2f51ced780:  0000000000000000  0000000000000000 
00007f2f51ced790:  0000001300000000  0000000000000120 
00007f2f51ced7a0:  0000000000000000  0000000000000000 
00007f2f51ced7b0:  0000000000000000  0000000000000000 
00007f2f51ced7c0:  0000000000000004  0000003400000013 
00007f2f51ced7d0:  0000000000000000  00007f2f78896a6e 
00007f2f51ced7e0: <0000000000000000  000000770000007c 
00007f2f51ced7f0:  0000005b0000006e  0000000000000000 
00007f2f51ced800:  0000000000000000  00007f2f78926a51 
00007f2f51ced810:  00007f2f437fe640  00007f2f51cedaf0 
00007f2f51ced820:  00007f2f51ced96e  00007f2f51ced96f 
00007f2f51ced830:  0000000000000000  00007f2f78894759 
00007f2f51ced840:  00000000007fff00  0000000000000000 
00007f2f51ced850:  00000000003d0f00  00007f2f437fe910 
00007f2f51ced860:  00007f2f437fe910  951972223d314e00 
00007f2f51ced870:  00007f2f51cee640  0000000000000006 
00007f2f51ced880:  0000000009ca52e0  0000000000000000 
00007f2f51ced890:  00000000058d3c90  00007f2f78842476 
00007f2f51ced8a0:  00007f2f78a1ae90  00007f2f788287f3 
00007f2f51ced8b0:  0000000000000020  00007f2f437fe640 
00007f2f51ced8c0:  0000000000000000  0000000000000001 
00007f2f51ced8d0:  00007f2f437fe640  00007f2f788955c4 

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc0007e24b8)
        /usr/local/go/src/runtime/sema.go:56 +0x42
sync.(*WaitGroup).Wait(0xc0007e24b0)
        /usr/local/go/src/sync/waitgroup.go:130 +0x64
github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run(0xc0001669a0, 0xc0007fa000, 0xc000c438d0, 0x1)
        /go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:258 +0x36e
github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc0007fa000, 0x536712f, 0xa, 0x536712f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:456 +0x795
github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0x536712f, 0xa, 0x536712f, 0xa, 0x0, 0x0, 0xc000c43c38, 0xc00045b2a0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189 +0x55a
github.com/elastic/beats/v7/libbeat/cmd/instance.Run(0x536712f, 0xa, 0x536712f, 0xa, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
        /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190 +0x148
github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc00046d080, 0xc000616800, 0x0, 0x8)
        /go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x9d
github.com/spf13/cobra.(*Command).execute(0xc00046d080, 0xc00004e0a0, 0x8, 0x8, 0xc00046d080, 0xc00004e0a0)
        /go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:766 +0x29d
github.com/spf13/cobra.(*Command).ExecuteC(0xc00046d080, 0x0, 0x0, 0xc0000a2058)
        /go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:852 +0x2ea
github.com/spf13/cobra.(*Command).Execute(...)
        /go/pkg/mod/github.com/spf13/cobra@v0.0.3/command.go:800
main.main()
        /go/src/github.com/elastic/beats/x-pack/metricbeat/main.go:21 +0x2f

goroutine 19 [select]:
github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor.func1(0xc0005442d0, 0xc000544190)
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:245 +0xba
created by github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:243 +0x92

goroutine 20 [select]:
github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor.func1(0xc000544320, 0xc0005441e0)
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:245 +0xba
created by github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:243 +0x92

goroutine 16 [chan receive]:
k8s.io/klog.(*loggingT).flushDaemon(0x7f571a0)
        /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:1010 +0x8b
created by k8s.io/klog.init.0
        /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:411 +0xd6

goroutine 21 [select]:
github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor.func1(0xc000544370, 0xc000544230)
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:245 +0xba
created by github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:243 +0x92

goroutine 22 [select]:
github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor.func1(0xc0005443c0, 0xc000544280)
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:245 +0xba
created by github.com/elastic/beats/v7/libbeat/common.(*Cache).StartJanitor
        /go/src/github.com/elastic/beats/libbeat/common/cache.go:243 +0x92

goroutine 23 [select]:
go.opencensus.io/stats/view.(*worker).start(0xc0005bc0a0)
        /go/pkg/mod/go.opencensus.io@v0.22.2/stats/view/worker.go:154 +0x100
created by go.opencensus.io/stats/view.init.0
        /go/pkg/mod/go.opencensus.io@v0.22.2/stats/view/worker.go:32 +0x57

goroutine 40 [chan receive]:
github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.(*cache).cleanup(0xc00100a520)
        /go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/cache.go:71 +0x186
created by github.com/elastic/beats/v7/libbeat/processors/add_kubernetes_metadata.newCache
        /go/src/github.com/elastic/beats/libbeat/processors/add_kubernetes_metadata/cache.go:42 +0x98

goroutine 1140 [select]:
github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc0007f37a0)
        /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:129 +0x3cb
github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1(0xc0007f37a0)
        /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:107 +0x50
created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter
        /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:105 +0x157

goroutine 44 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc000592960)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x1cf
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func1(0xc000166460, 0x59a4dc0, 0xc000592960)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:171 +0x5f
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:169 +0x373

goroutine 45 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc0000980f0)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x107
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2(0xc000166460, 0xc0000980f0)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:175 +0x59
created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:173 +0x3a2

goroutine 46 [select]:
github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc00100b700, 0x32, 0x0, 0x0, 0x0, 0x0)
        /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:65 +0xe9
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc0004d8f00, 0x59a4e00, 0xc00100b700)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:182 +0x19a
github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1(0xc0004d8f00, 0x59a4e00, 0xc00100b6e0)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:86 +0x6c
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:84 +0x149

goroutine 47 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*retryer).loop(0xc0004d9020)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:135 +0x250
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newRetryer
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:94 +0x158

goroutine 48 [select]:
github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0004d9200)
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:127 +0xd1
created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
        /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x194

goroutine 1141 [syscall]:
os/signal.signal_recv(0x54de848)
        /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
os/signal.loop()
        /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
created by os/signal.Notify.func1
        /usr/local/go/src/os/signal/signal.go:127 +0x44

goroutine 1143 [chan receive]:
github.com/elastic/beats/v7/libbeat/service.HandleSignals.func1(0xc00085aae0, 0xc00085c670, 0xc00085c660, 0xc0007e2470, 0xc0007dd5c0)
        /go/src/github.com/elastic/beats/libbeat/service/service.go:50 +0x52
created by github.com/elastic/beats/v7/libbeat/service.HandleSignals
        /go/src/github.com/elastic/beats/libbeat/service/service.go:49 +0x172

goroutine 1145 [chan receive]:
github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run.func2(0xc0007e24b0, 0xc0001669a0, 0xc00063ecd0)
        /go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:226 +0x63
created by github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run
        /go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:224 +0x32c

goroutine 1146 [select]:
github.com/elastic/beats/v7/libbeat/cfgfile.(*Reloader).Run(0xc00085ad20, 0x59a5140, 0xc0005934a0)
        /go/src/github.com/elastic/beats/libbeat/cfgfile/reload.go:188 +0x229
created by github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run
        /go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:238 +0x492

goroutine 1147 [chan receive]:
github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run.func3(0xc0007e24b0, 0xc0001669a0, 0xc00085ad20)
        /go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:242 +0x63
created by github.com/elastic/beats/v7/metricbeat/beater.(*Metricbeat).Run
        /go/src/github.com/elastic/beats/metricbeat/beater/metricbeat.go:240 +0x4e8

rax    0x0
rbx    0x7f2f51cee640
rcx    0x7f2f78896a7c
rdx    0x6
rdi    0x854
rsi    0x855
rbp    0x855
rsp    0x7f2f51ced7e0
r8     0x7f2f51ced8b0
r9     0x7fffffff
r10    0x8
r11    0x246
r12    0x6
r13    0x16
r14    0x58d3c90
r15    0x0
rip    0x7f2f78896a7c
rflags 0x246
cs     0x33
fs     0x0
gs     0x0

Hi,

I found a similar issue here regarding the apm-server, so you could try this workaround first and disable seccomp:

seccomp:
  enabled: false

Best regards
Wolfram

In metricbeat.yml file?

Yes, in the metricbeat config.

Thanks.

May i know what the actual root cause of this issue? Will this workaround still needed if I use the latest versions?

I am not sure what the root cause is - we are running Metricbeat on linux too and did not have this issue.

Thanks. Its working.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.