Filebeat irregularly restarts and crashes

aleaesaest · August 10, 2022, 10:37am

Hello,

we are using wazuh installed on a VM.

Unfortunately, the filebeat service crashes at irregular intervals - after running for a while.

I reviewed all the logs I could, but was unable to find the cause of the issue.

I would be really thankful for any advice.

 systemctl status filebeat
× filebeat.service - Filebeat sends log files to Logstash or directly to Elasti>
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor pres>
     Active: failed (Result: exit-code) since Wed 2022-08-10 11:27:16 CEST; 1h >
       Docs: https://www.elastic.co/products/beats/filebeat
    Process: 6312 ExecStart=/usr/share/filebeat/bin/filebeat --environment syst>
   Main PID: 6312 (code=exited, status=2)
        CPU: 49ms

Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: rip    0x7ff339816a7c
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: rflags 0x246
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: cs     0x33
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: fs     0x0
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: gs     0x0
Aug 10 11:27:16 wazuh-srv01 systemd[1]: filebeat.service: Scheduled restart job>
Aug 10 11:27:16 wazuh-srv01 systemd[1]: Stopped Filebeat sends log files to Log>
Aug 10 11:27:16 wazuh-srv01 systemd[1]: filebeat.service: Start request repeate>
Aug 10 11:27:16 wazuh-srv01 systemd[1]: filebeat.service: Failed with result 'e>
Aug 10 11:27:16 wazuh-srv01 systemd[1]: Failed to start Filebeat sends log file>
lines 1-18/18 (END)...skipping...
× filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
     Loaded: loaded (/lib/systemd/system/filebeat.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Wed 2022-08-10 11:27:16 CEST; 1h 7min ago
       Docs: https://www.elastic.co/products/beats/filebeat
    Process: 6312 ExecStart=/usr/share/filebeat/bin/filebeat --environment systemd $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=2)
   Main PID: 6312 (code=exited, status=2)
        CPU: 49ms

Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: rip    0x7ff339816a7c
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: rflags 0x246
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: cs     0x33
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: fs     0x0
Aug 10 11:27:16 wazuh-srv01 filebeat[6312]: gs     0x0
Aug 10 11:27:16 wazuh-srv01 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 20.
Aug 10 11:27:16 wazuh-srv01 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
Aug 10 11:27:16 wazuh-srv01 systemd[1]: filebeat.service: Start request repeated too quickly.
Aug 10 11:27:16 wazuh-srv01 systemd[1]: filebeat.service: Failed with result 'exit-code'.
Aug 10 11:27:16 wazuh-srv01 systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..

79g · August 10, 2022, 10:48am

Hello,

if you paste a significant amount of Filebeat logs, maybe we could help you.

Paste the log instead of the status output.

aleaesaest · August 10, 2022, 11:02am

Thanks for your reply. Here is the latest log I get from journalctl -xe -u filebeat

Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/spf13/cobra.(*Command).Execute(...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: main.main()
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/filebeat/main.go:36 +0x3d
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 62 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc0005ca320)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x1cf
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func1(0xc0001e6770, 0x2d3ed60, 0xc0005ca320)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:171 +0x5f
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
Aug 10 12:38:37 wazuh-srv01 systemd[1]: filebeat.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit filebeat.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 2.
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:169 +0x373
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 7 [chan receive]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: k8s.io/klog.(*loggingT).flushDaemon(0x4241ce0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:1010 +0x8b
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by k8s.io/klog.init.0
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:411 +0xd6
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 63 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc0000a00f0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x107
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2(0xc0001e6770, 0xc0000a00f0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:175 +0x59
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:173 +0x3a2
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 64 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc0005b1a40, 0x32, 0x0, 0x0, 0x0, 0x0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:65 +0xe9
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc0005a44e0, 0x2d3eda0, 0xc0005b1a40)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:182 +0x19a
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1(0xc0005a44e0, 0x2d3eda0, 0xc0005b1a20)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:86 +0x6c
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:84 +0x149
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 65 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*retryer).loop(0xc0005a4600)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:135 +0x250
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newRetryer
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:94 +0x158
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 66 [select]:
Aug 10 12:38:37 wazuh-srv01 systemd[1]: filebeat.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit filebeat.service has entered the 'failed' state with result 'exit-code'.
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0005a4840)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:127 +0xd1
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x194
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 89 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc0007d8f60)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:129 +0x3cb
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1(0xc0007d8f60)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:107 +0x50
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:105 +0x157
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 90 [syscall]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: os/signal.signal_recv(0x0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: os/signal.loop()
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by os/signal.Notify.func1
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /usr/local/go/src/os/signal/signal.go:127 +0x44
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 92 [chan receive]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/service.HandleSignals.func1(0xc0005e4f60, 0xc000349d50, 0xc000349d40, 0xc000125db0, 0xc0004e6ee0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/service/service.go:50 +0x52
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/service.HandleSignals
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/service/service.go:49 +0x172
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: goroutine 93 [runnable]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: github.com/elastic/beats/v7/libbeat/service.ProcessWindowsControlEvents(0xc0004e6f00)
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/service/service_unix.go:24
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: created by github.com/elastic/beats/v7/libbeat/service.HandleSignals
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]:         /go/src/github.com/elastic/beats/libbeat/service/service.go:64 +0x1d5
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rax    0x0
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rbx    0x7ff84966d640
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rcx    0x7ff872dbca7c
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rdx    0x6
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rdi    0x1c11
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rsi    0x1c18
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rbp    0x1c18
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rsp    0x7ff84966c7b0
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r8     0x7ff84966c880
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r9     0x7fffffff
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r10    0x8
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r11    0x246
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r12    0x6
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r13    0x16
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r14    0x2cf0118
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: r15    0x0
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rip    0x7ff872dbca7c
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: rflags 0x246
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: cs     0x33
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: fs     0x0
Aug 10 12:38:37 wazuh-srv01 filebeat[7185]: gs     0x0
Aug 10 12:38:37 wazuh-srv01 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 1.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit filebeat.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Aug 10 12:38:37 wazuh-srv01 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
░░ Subject: A stop job for unit filebeat.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit filebeat.service has finished.
░░
░░ The job identifier is 6238 and the job result is done.
Aug 10 12:38:37 wazuh-srv01 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
░░ Subject: A start job for unit filebeat.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit filebeat.service has finished successfully.
░░
░░ The job identifier is 6238.
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: runtime/cgo: pthread_create failed: Operation not permitted
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: SIGABRT: abort
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: PC=0x7f6025284a7c m=4 sigcode=18446744073709551610
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 0 [idle]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: runtime: unknown pc 0x7f6025284a7c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: stack: frame={sp:0x7f5ffdd587b0, fp:0x0} stack=[0x7f5ffd5591e8,0x7f5ffdd58de8)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586b0:  0000000004014f40  0000000003002878
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586c0:  00007f5ffdd58750  0000000000ee923a <runtime.scanframeworker+122>
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586d0:  00007f5ffdd589f8  00007f5ffdd58b08
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586e0:  0000000002af4800  000000c000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586f0:  0000000000000000  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58700:  0000000000000000  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58710:  0000000000000000  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58720:  0000000004014f40  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58730:  0000000000f32ac1 <runtime.goexit+1>  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58740:  0000000000000000  0000000000000130
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58750:  00007f5ffdd58778  0000000000f2dbdd <runtime.scanstack.func1+61>
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58760:  00000013fdd589f8  0000000000000120
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58770:  000000c00005fe98  00007f5ffdd58a50
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58780:  0000000000f269ee <runtime.gentraceback+4366>  00007f5ffdd589f8
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58790:  0000000000000004  0000003400000013
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587a0:  0000000000000000  00007f6025284a6e
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587b0: <0000000000000000  000000770000007c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587c0:  0000005b0000006e  0000000000000001
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587d0:  0000000000000000  00007f6025314a51
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587e0:  00007f5ff6ffd640  00007f5ffdd58ac0
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587f0:  00007f5ffdd5893e  00007f5ffdd5893f
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58800:  0000000000000000  00007f6025282759
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58810:  00000000007fff00  0000000000f1afdb <runtime.getStackMap+395>
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58820:  00000000003d0f00  00007f5ff6ffd910
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58830:  00007f5ff6ffd910  f004e50d2e9c2900
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58840:  00007f5ffdd59640  0000000000000006
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58850:  00007f5fe8000ca0  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58860:  0000000002cf0118  00007f6025230476
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58870:  00007f6025408e90  00007f60252167f3
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58880:  0000000000000020  00007f5ff6ffd640
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58890:  0000000000000000  0000000000000001
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd588a0:  00007f5ff6ffd640  00007f60252835c4
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: runtime: unknown pc 0x7f6025284a7c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: stack: frame={sp:0x7f5ffdd587b0, fp:0x0} stack=[0x7f5ffd5591e8,0x7f5ffdd58de8)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586b0:  0000000004014f40  0000000003002878
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586c0:  00007f5ffdd58750  0000000000ee923a <runtime.scanframeworker+122>
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586d0:  00007f5ffdd589f8  00007f5ffdd58b08
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586e0:  0000000002af4800  000000c000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd586f0:  0000000000000000  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58700:  0000000000000000  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58710:  0000000000000000  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58720:  0000000004014f40  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58730:  0000000000f32ac1 <runtime.goexit+1>  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58740:  0000000000000000  0000000000000130
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58750:  00007f5ffdd58778  0000000000f2dbdd <runtime.scanstack.func1+61>
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58760:  00000013fdd589f8  0000000000000120
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58770:  000000c00005fe98  00007f5ffdd58a50
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58780:  0000000000f269ee <runtime.gentraceback+4366>  00007f5ffdd589f8
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58790:  0000000000000004  0000003400000013
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587a0:  0000000000000000  00007f6025284a6e
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587b0: <0000000000000000  000000770000007c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587c0:  0000005b0000006e  0000000000000001
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587d0:  0000000000000000  00007f6025314a51
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587e0:  00007f5ff6ffd640  00007f5ffdd58ac0
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd587f0:  00007f5ffdd5893e  00007f5ffdd5893f
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58800:  0000000000000000  00007f6025282759
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58810:  00000000007fff00  0000000000f1afdb <runtime.getStackMap+395>
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58820:  00000000003d0f00  00007f5ff6ffd910
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58830:  00007f5ff6ffd910  f004e50d2e9c2900
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58840:  00007f5ffdd59640  0000000000000006
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58850:  00007f5fe8000ca0  0000000000000000
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58860:  0000000002cf0118  00007f6025230476
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58870:  00007f6025408e90  00007f60252167f3
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58880:  0000000000000020  00007f5ff6ffd640
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd58890:  0000000000000000  0000000000000001
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: 00007f5ffdd588a0:  00007f5ff6ffd640  00007f60252835c4
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 1 [runnable]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: os/signal.signal_enable(0x14bc09e300000001)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /usr/local/go/src/runtime/sigqueue.go:219 +0x6c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: os/signal.enableSignal(...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /usr/local/go/src/os/signal/signal_unix.go:51
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: os/signal.Notify.func2(0x1)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /usr/local/go/src/os/signal/signal.go:150 +0x8e
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: os/signal.Notify(0xc0000aec60, 0xc0007ef3b8, 0x3, 0x3)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /usr/local/go/src/os/signal/signal.go:162 +0x170
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/service.HandleSignals(0xc000403220, 0xc0004aa950)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/service/service.go:48 +0x122
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/cmd/instance.(*Beat).launch(0xc00014c2c0, 0x29fa2ec, 0x8, 0x29fa2ec, 0x8, 0x0, 0x0, 0x1, 0x0, 0x0, ...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:448 +0x633
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1(0x29fa2ec, 0x8, 0x29fa2ec, 0x8, 0x0, 0x0, 0xc00041fc30, 0xc0002a74c0, 0x0, 0x0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:189 +0x55a
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/cmd/instance.Run(0x29fa2ec, 0x8, 0x29fa2ec, 0x8, 0x0, 0x0, 0x1, 0x0, 0x0, 0x0, ...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/cmd/instance/beat.go:190 +0x148
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/cmd.genRunCmd.func1(0xc00036a2c0, 0xc0000b0180, 0x0, 0xc)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/cmd/run.go:36 +0x9d
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/spf13/cobra.(*Command).execute(0xc00036a2c0, 0xc00011e0e0, 0xc, 0xc, 0xc00036a2c0, 0xc00011e0e0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:830 +0x29d
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/spf13/cobra.(*Command).ExecuteC(0xc00036a2c0, 0xc00036a2c0, 0xffffffff, 0xc00010e058)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:914 +0x2fb
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/spf13/cobra.(*Command).Execute(...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/github.com/spf13/cobra@v0.0.5/command.go:864
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: main.main()
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/filebeat/main.go:36 +0x3d
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 10 [chan receive]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: k8s.io/klog.(*loggingT).flushDaemon(0x4241ce0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:1010 +0x8b
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by k8s.io/klog.init.0
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/k8s.io/klog@v1.0.0/klog.go:411 +0xd6
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 74 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*bufferingEventLoop).run(0xc000296be0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/eventloop.go:316 +0x1cf
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func1(0xc0001ec4d0, 0x2d3ed60, 0xc000296be0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:171 +0x5f
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:169 +0x373
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 75 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*ackLoop).run(0xc00015b400)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/ackloop.go:60 +0x107
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue.func2(0xc0001ec4d0, 0xc00015b400)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:175 +0x59
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.NewQueue
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/broker.go:173 +0x3a2
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 76 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/queue/memqueue.(*consumer).Get(0xc00000e9e0, 0x32, 0x0, 0x0, 0x0, 0x0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/queue/memqueue/consume.go:65 +0xe9
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*eventConsumer).loop(0xc0004d97a0, 0x2d3eda0, 0xc00000e9e0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:182 +0x19a
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer.func1(0xc0004d97a0, 0x2d3eda0, 0xc00000e9c0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:86 +0x6c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newEventConsumer
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/consumer.go:84 +0x149
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 77 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*retryer).loop(0xc0004d98c0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:135 +0x250
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.newRetryer
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/retry.go:94 +0x158
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 78 [select]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/publisher/pipeline.(*netClientWorker).run(0xc0004d9aa0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:127 +0xd1
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by github.com/elastic/beats/v7/libbeat/publisher/pipeline.makeClientWorker
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/publisher/pipeline/output.go:79 +0x194
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 91 [runnable]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: go.uber.org/zap/zapcore.CapitalLevelEncoder(0xc0b5008b51a68000, 0x2db0880, 0xc00046c020)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/go.uber.org/zap@v1.14.0/zapcore/encoder.go:58 +0x4d
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: go.uber.org/zap/zapcore.consoleEncoder.EncodeEntry(0xc0006d1a70, 0x0, 0xc0b5008b51a68047, 0x1fb0ec2, 0x42417a0, 0x29fdff7, 0xa, 0xc000466000, 0x22, 0x1, ...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/go.uber.org/zap@v1.14.0/zapcore/console_encoder.go:80 +0x895
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: go.uber.org/zap/zapcore.(*ioCore).Write(0xc0006d1aa0, 0x0, 0xc0b5008b51a68047, 0x1fb0ec2, 0x42417a0, 0x29fdff7, 0xa, 0xc000466000, 0x22, 0x1, ...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/go.uber.org/zap@v1.14.0/zapcore/core.go:86 +0xa9
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: go.uber.org/zap/zapcore.(*CheckedEntry).Write(0xc00046a000, 0x0, 0x0, 0x0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/go.uber.org/zap@v1.14.0/zapcore/entry.go:216 +0x117
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: go.uber.org/zap.(*SugaredLogger).log(0xc000010320, 0xc000124000, 0x2a2c102, 0x21, 0xc000587e40, 0x1, 0x1, 0x0, 0x0, 0x0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/go.uber.org/zap@v1.14.0/sugar.go:234 +0x100
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: go.uber.org/zap.(*SugaredLogger).Infof(...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/pkg/mod/go.uber.org/zap@v1.14.0/sugar.go:138
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/logp.(*Logger).Infof(...)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/logp/logger.go:119
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/monitoring/report/log.(*reporter).snapshotLoop(0xc0004b04b0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:118 +0xfd
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter.func1(0xc0004b04b0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:107 +0x50
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by github.com/elastic/beats/v7/libbeat/monitoring/report/log.MakeReporter
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /go/src/github.com/elastic/beats/libbeat/monitoring/report/log/log.go:105 +0x157
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: goroutine 92 [syscall]:
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: os/signal.signal_recv(0x0)
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /usr/local/go/src/runtime/sigqueue.go:147 +0x9c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: os/signal.loop()
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /usr/local/go/src/os/signal/signal_unix.go:23 +0x22
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: created by os/signal.Notify.func1
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]:         /usr/local/go/src/os/signal/signal.go:127 +0x44
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rax    0x0
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rbx    0x7f5ffdd59640
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rcx    0x7f6025284a7c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rdx    0x6
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rdi    0x1c1a
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rsi    0x1c1d
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rbp    0x1c1d
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rsp    0x7f5ffdd587b0
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r8     0x7f5ffdd58880
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r9     0x7fffffff
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r10    0x8
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r11    0x246
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r12    0x6
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r13    0x16
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r14    0x2cf0118
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: r15    0x0
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rip    0x7f6025284a7c
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: rflags 0x246
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: cs     0x33
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: fs     0x0
Aug 10 12:38:37 wazuh-srv01 filebeat[7194]: gs     0x0
Aug 10 12:38:37 wazuh-srv01 systemd[1]: filebeat.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ An ExecStart= process belonging to unit filebeat.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 2.
Aug 10 12:38:37 wazuh-srv01 systemd[1]: filebeat.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ The unit filebeat.service has entered the 'failed' state with result 'exit-code'.
Aug 10 12:38:37 wazuh-srv01 systemd[1]: filebeat.service: Scheduled restart job, restart counter is at 2.
░░ Subject: Automatic restarting of a unit has been scheduled
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ Automatic restarting of the unit filebeat.service has been scheduled, as the result for
░░ the configured Restart= setting for the unit.
Aug 10 12:38:37 wazuh-srv01 systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
░░ Subject: A stop job for unit filebeat.service has finished
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A stop job for unit filebeat.service has finished.
░░
░░ The job identifier is 6322 and the job result is done.
Aug 10 12:38:37 wazuh-srv01 systemd[1]: Started Filebeat sends log files to Logstash or directly to Elasticsearch..
░░ Subject: A start job for unit filebeat.service has finished successfully
░░ Defined-By: systemd
░░ Support: http://www.ubuntu.com/support
░░
░░ A start job for unit filebeat.service has finished successfully.
░░
░░ The job identifier is 6322.

aleaesaest · August 11, 2022, 7:38am

(I edited the topic and moved it to filebeat)

cheshirecat · August 11, 2022, 11:17am

Had same issue with my ELK Stack on Debian 11. The thing is that your Filebeat instance is flooding output (Logstash i guess). Please try to change values for first intervals (300h -> 3h, etc.).

BR,
Chris

aleaesaest · August 11, 2022, 11:40am

Thank you very much for your feedback! I will be happy to try that - where exactly can I adjust that?

cheshirecat · August 11, 2022, 11:45am

Look at the file:

/etc/filebeat/modules.d/threatintel.yml

search for:

    var.first_interval:

aleaesaest · August 11, 2022, 11:53am

No such file exists in the folder:

/etc/filebeat/modules.d# dir
apache.yml.disabled icinga.yml.disabled logstash.yml.disabled nginx.yml.disabled santa.yml.disabled
auditd.yml.disabled iis.yml.disabled mongodb.yml.disabled osquery.yml.disabled system.yml.disabled
elasticsearch.yml.disabled kafka.yml.disabled mysql.yml.disabled postgresql.yml.disabled traefik.yml.disabled
haproxy.yml.disabled kibana.yml.disabled nats.yml.disabled redis.yml.disabled

cheshirecat · August 11, 2022, 12:19pm

This is one of filebeats modules:

I use 7.17 version.

aleaesaest · August 11, 2022, 12:48pm

Am still using filebeat version 7.10.4
Seems that threatintel.yml is not included here.

Do you have any suggestion about other modules fitting here?

cheshirecat · August 11, 2022, 1:01pm

Could you please list enabled filebeat modules?

filebeat modules list

aleaesaest · August 11, 2022, 1:03pm

/etc/filebeat# filebeat modules list
Enabled:

Disabled:
apache
auditd
elasticsearch
haproxy
icinga
iis
kafka
kibana
logstash
mongodb
mysql
nats
nginx
osquery
postgresql
redis
santa
system
traefik

cheshirecat · August 11, 2022, 1:31pm

As you have non of modules enabled, could you please post filebeat.yml?

aleaesaest · August 11, 2022, 1:33pm

I was just about to do that
Here you go:

# Wazuh - Filebeat configuration file
output.elasticsearch:
  protocol: https
  username: ${username}
  password: ${password}
  ssl.certificate_authorities:
    - /etc/filebeat/certs/root-ca.pem
  ssl.certificate: "/etc/filebeat/certs/WAZUH-SRV01.pem"
  ssl.key: "/etc/filebeat/certs/WAZUH-SRV01-key.pem"
setup.template.json.enabled: true
setup.template.json.path: '/etc/filebeat/wazuh-template.json'
setup.template.json.name: 'wazuh'
setup.ilm.overwrite: true
setup.ilm.enabled: false

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false


logging.level: info
logging.to_files: true
logging.files:
  path: /var/log/filebeat
  name: filebeat
  keepfiles: 7
  permissions: 0644

output.elasticsearch.hosts:
  - "192.168.194.81:9200"

seccomp:
default_action: allow
syscalls:
- action: allow
  names:
  - rseq
  - clone3

aleaesaest · August 11, 2022, 2:16pm

I just added "max_procs: 2" so that filebeat is only allowed to start 2 threads.
Seems that filebeat is running again.

Am not sure if that is an intended solution.

cheshirecat · August 11, 2022, 7:45pm

Is it still working?

aleaesaest · August 12, 2022, 5:43am

Yes, it is still running the next morning. Yai!

Seems that filebeat was trying to use more cores than allowed.

Thank you very much for your assistance!

Do you have any suggestions on my config?

cheshirecat · August 12, 2022, 5:52am

Hello,
In your config file we can see:

output.elasticsearch.hosts:
  - "192.168.194.81:9200"

Please don't use default ports, for security reasons. Elasticsearch, Filebeat, Kibana (and so on...) can use any port. Maybe you should consider port number above 10000 for ES? You have just remember to use same port in all config files (and if you have more than one node Ansible can be useful).

For modules You use inline configuration:

filebeat.modules:
  - module: wazuh
    alerts:
      enabled: true
    archives:
      enabled: false

you can create a new module config file:

nano /etc/filebeat/modules.d/wazuh.yml

and move configuration to that file.
Than you will be able to use commands:

filebeat modules enable|disable wuzah

if needed.

BR,
Chris

aleaesaest · August 12, 2022, 9:32am

Thank you very much for your assistance and advice!

system · September 9, 2022, 9:32am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Help. My filebeat stops working after I installed my wazuh server after 30 minutes Logstash	2	433	March 12, 2023
Filebeat is down after start Beats filebeat	3	636	March 31, 2022
Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch Beats filebeat	2	288	November 30, 2022
Filebeat failed to start v7.10.2 Beats filebeat	2	335	April 14, 2023
Filebeat stopped working after an hour after the install Logstash	2	226	March 6, 2023

Filebeat irregularly restarts and crashes

Related Topics