File beat stops time to time in RHEL environment

ruranga · January 6, 2020, 9:06am

Hi,

I have installed filbeat in RHEL syslog server and sending the logs to ELK stack at another server, By the way filebeat in the syslog server getting stopped time to time. When I check the logs following found

2020-01-06T07:33:11.233Z INFO beater/filebeat.go:443 Stopping filebeat
2020-01-06T07:33:11.292Z INFO crawler/crawler.go:139 Stopping Crawler
2020-01-06T07:33:11.430Z INFO crawler/crawler.go:149 Stopping 1 inputs
2020-01-06T07:33:11.430Z INFO input/input.go:149 input ticker stopped
2020-01-06T07:33:11.430Z INFO cfgfile/reload.go:229 Dynamic config reloader stopped
2020-01-06T07:33:11.430Z INFO input/input.go:167 Stopping Input: 6167974775168247055
2020-01-06T07:33:11.431Z INFO log/harvester.go:272 Reader was closed: /logs/DC1-Proxy-GW3/dc1-proxy-gw3_proxy.log. Closing.
2020-01-06T07:33:11.431Z INFO crawler/crawler.go:165 Crawler stopped
2020-01-06T07:33:11.431Z INFO registrar/registrar.go:367 Stopping Registrar
2020-01-06T07:33:11.431Z INFO registrar/registrar.go:293 Ending Registrar
2020-01-06T07:33:11.440Z INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":66700,"time":{"ms":66703}},"total":{"ticks":496080,"time":{"ms":496085},"value":496080},"user":{"ticks":429380,"time":{"ms":429382}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":7},"info":{"ephemeral_id":"6246d6ce-83f1-4636-836e-4946c73d8f84","uptime":{"ms":8381436}},"memstats":{"gc_next":16522000,"memory_alloc":8437704,"memory_total":34548886184,"rss":35966976},"runtime":{"goroutines":14}},"filebeat":{"events":{"active":386,"added":4189851,"done":4189465},"harvester":{"closed":3,"open_files":0,"running":0,"started":3},"input":{"log":{"files":{"truncated":2}}}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":4189459,"batches":8489,"total":4189459},"read":{"bytes":50934},"type":"logstash","write":{"bytes":259049178}},"pipeline":{"clients":0,"events":{"active":386,"filtered":6,"published":4189845,"retry":2048,"total":4189851},"queue":{"acked":4189459}}},"registrar":{"states":{"current":2,"update":4189465},"writes":{"success":8484,"total":8484}},"system":{"cpu":{"cores":8},"load":{"1":1.96,"15":1.15,"5":1.36,"norm":{"1":0.245,"15":0.1438,"5":0.17}}}}}}
2020-01-06T07:33:11.441Z INFO [monitoring] log/log.go:154 Uptime: 2h19m41.441602094s
2020-01-06T07:33:11.441Z INFO [monitoring] log/log.go:131 Stopping metrics logging.
2020-01-06T07:33:11.441Z INFO instance/beat.go:435 filebeat stopped.

Can someone pls help!
Thanks in advance!

KInd regards

Jugsofbeer · January 7, 2020, 6:29am

Is there any info before "2020-01-06T07:33:11.233Z INFO beater/filebeat.go:443 Stopping filebeat".

Perhaps you have full disks?
maybe something on cron thats stopping filebeat?

ruranga · January 20, 2020, 9:36am

Dear jugsofbeer,

I'm still getting the issue! Pls find the logs that I have captured from the filebeat logs regarding the unexpected stop.

2020-01-17T13:50:12.117Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":27120,"time":{"ms":40}},"total":{"ticks":174770,"time":{"ms":199},"value":174770},"user":{"ticks":147650,"time":{"ms":159}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"11688eeb-7a4c-478a-8aa6-29b555d61d0d","uptime":{"ms":9211109}},"memstats":{"gc_next":12126544,"memory_alloc":7707064,"memory_total":17357756912},"runtime":{"goroutines":27}},"filebeat":{"events":{"active":-93,"added":1747,"done":1840},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1840,"batches":15,"total":1840},"read":{"bytes":90},"write":{"bytes":133010}},"pipeline":{"clients":1,"events":{"active":0,"published":1747,"total":1747},"queue":{"acked":1840}}},"registrar":{"states":{"current":1,"update":1840},"writes":{"success":15,"total":15}},"system":{"load":{"1":0.65,"15":0.67,"5":0.59,"norm":{"1":0.0813,"15":0.0838,"5":0.0738}}}}}}
2020-01-17T13:50:42.117Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":27180,"time":{"ms":52}},"total":{"ticks":175030,"time":{"ms":262},"value":175030},"user":{"ticks":147850,"time":{"ms":210}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"11688eeb-7a4c-478a-8aa6-29b555d61d0d","uptime":{"ms":9241109}},"memstats":{"gc_next":11522128,"memory_alloc":5887048,"memory_total":17389379224},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":1837,"done":1837},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1837,"batches":14,"total":1837},"read":{"bytes":84},"write":{"bytes":140766}},"pipeline":{"clients":1,"events":{"active":0,"published":1837,"total":1837},"queue":{"acked":1837}}},"registrar":{"states":{"current":1,"update":1837},"writes":{"success":14,"total":14}},"system":{"load":{"1":0.73,"15":0.68,"5":0.61,"norm":{"1":0.0913,"15":0.085,"5":0.0763}}}}}}
2020-01-17T13:51:12.117Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":27240,"time":{"ms":62}},"total":{"ticks":175340,"time":{"ms":307},"value":175340},"user":{"ticks":148100,"time":{"ms":245}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"11688eeb-7a4c-478a-8aa6-29b555d61d0d","uptime":{"ms":9271109}},"memstats":{"gc_next":11467728,"memory_alloc":8437832,"memory_total":17423366936},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":1729,"done":1729},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1729,"batches":16,"total":1729},"read":{"bytes":96},"write":{"bytes":132044}},"pipeline":{"clients":1,"events":{"active":0,"published":1729,"total":1729},"queue":{"acked":1729}}},"registrar":{"states":{"current":1,"update":1729},"writes":{"success":16,"total":16}},"system":{"load":{"1":0.81,"15":0.69,"5":0.65,"norm":{"1":0.1013,"15":0.0862,"5":0.0813}}}}}}
2020-01-17T13:51:42.119Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":27290,"time":{"ms":53}},"total":{"ticks":175630,"time":{"ms":295},"value":175630},"user":{"ticks":148340,"time":{"ms":242}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":8},"info":{"ephemeral_id":"11688eeb-7a4c-478a-8aa6-29b555d61d0d","uptime":{"ms":9301111}},"memstats":{"gc_next":11884880,"memory_alloc":8531752,"memory_total":17456618512},"runtime":{"goroutines":27}},"filebeat":{"events":{"active":110,"added":1846,"done":1736},"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":1736,"batches":15,"total":1736},"read":{"bytes":90},"write":{"bytes":128961}},"pipeline":{"clients":1,"events":{"active":110,"published":1846,"total":1846},"queue":{"acked":1736}}},"registrar":{"states":{"current":1,"update":1736},"writes":{"success":15,"total":15}},"system":{"load":{"1":0.77,"15":0.69,"5":0.65,"norm":{"1":0.0963,"15":0.0862,"5":0.0813}}}}}}
2020-01-17T13:51:42.334Z        INFO    beater/filebeat.go:443  Stopping filebeat
2020-01-17T13:51:42.334Z        INFO    crawler/crawler.go:139  Stopping Crawler
2020-01-17T13:51:42.334Z        INFO    crawler/crawler.go:149  Stopping 1 inputs
2020-01-17T13:51:42.334Z        INFO    cfgfile/reload.go:229   Dynamic config reloader stopped
2020-01-17T13:51:42.334Z        INFO    input/input.go:149      input ticker stopped
2020-01-17T13:51:42.334Z        INFO    input/input.go:167      Stopping Input: 6167974775168247055
2020-01-17T13:51:42.334Z        INFO    log/harvester.go:272    Reader was closed: /logs/DC1-Proxy-GW3/dc1-proxy-gw3_proxy.log. Closing.
2020-01-17T13:51:42.334Z        INFO    crawler/crawler.go:165  Crawler stopped
2020-01-17T13:51:42.335Z        INFO    registrar/registrar.go:367      Stopping Registrar
2020-01-17T13:51:42.335Z        INFO    registrar/registrar.go:293      Ending Registrar
2020-01-17T13:51:42.338Z        INFO    [monitoring]    log/log.go:153  Total non-zero metrics  {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":27290,"time":{"ms":27299}},"total":{"ticks":175640,"time":{"ms":175653},"value":175640},"user":{"ticks":148350,"time":{"ms":148354}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":7},"info":{"ephemeral_id":"11688eeb-7a4c-478a-8aa6-29b555d61d0d","uptime":{"ms":9301331}},"memstats":{"gc_next":11884880,"memory_alloc":8767560,"memory_total":17456854320,"rss":50941952},"runtime":{"goroutines":14}},"filebeat":{"events":{"active":110,"added":1140551,"done":1140441},"harvester":{"closed":4,"open_files":0,"running":0,"started":4},"input":{"log":{"files":{"truncated":3}}}},"libbeat":{"config":{"module":{"running":0},"reloads":1},"output":{"events":{"acked":1140433,"batches":7362,"total":1140433},"read":{"bytes":44178},"type":"logstash","write":{"bytes":84742834}},"pipeline":{"clients":0,"events":{"active":110,"filtered":8,"published":1140543,"retry":2048,"total":1140551},"queue":{"acked":1140433}}},"registrar":{"states":{"current":1,"update":1140441},"writes":{"success":7368,"total":7368}},"system":{"cpu":{"cores":8},"load":{"1":0.77,"15":0.69,"5":0.65,"norm":{"1":0.0963,"15":0.0862,"5":0.0813}}}}}}
2020-01-17T13:51:42.338Z        INFO    [monitoring]    log/log.go:154  Uptime: 2h35m1.332719683s
2020-01-17T13:51:42.338Z        INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2020-01-17T13:51:42.338Z        INFO    instance/beat.go:435    filebeat stopped.
[root@syslog logs]#

system · February 17, 2020, 9:36am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.