Hello team,
I am unable to start and stop filebeat . getting error like failed to stop filebeat.
Can you please help on this.
PFB filebeat logs:
2021-05-26T14:08:56.070-0400 INFO instance/beat.go:640 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-05-26T14:08:56.071-0400 INFO instance/beat.go:648 Beat ID: ce29b12d-0ea9-4044-9312-e3c9791201b8
2021-05-26T14:08:56.074-0400 INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp
2021-05-26T14:08:56.074-0400 INFO [beat] instance/beat.go:976 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "ce29b12d-0ea9-4044-9312-e3c9791201b8"}}}
2021-05-26T14:08:56.074-0400 INFO [beat] instance/beat.go:985 Build info {"system_info": {"build": {"commit": "2ab907f5ccecf9fd82fe37105082e89fd871f684", "libbeat": "7.9.2", "time": "2020-09-22T23:19:45.000Z", "version": "7.9.2"}}}
2021-05-26T14:08:56.074-0400 INFO [beat] instance/beat.go:988 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":40,"version":"go1.14.7"}}}
2021-05-26T14:08:56.075-0400 INFO [beat] instance/beat.go:992 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-04-07T23:30:21-04:00","containerized":false,"name":"WUWVC5ASGRD05.prod.wudip.com","ip":["127.0.0.1/8","10.45.16.205/22"],"kernel_version":"2.6.32-754.36.1.el6.x86_64","mac":["00:17:a4:77:38:30","00:17:a4:77:38:30","00:17:a4:77:38:2e","00:17:a4:77:38:2f","00:17:a4:77:38:30"],"os":{"family":"redhat","platform":"redhat","name":"Red","version":"6.10 (Santiago)","major":6,"minor":10,"patch":0,"codename":"Santiago"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"b65197304d2cb376f829ef3f00000011"}}}
2021-05-26T14:08:56.076-0400 INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.9.2
2021-05-26T14:08:56.079-0400 INFO [publisher] pipeline/module.go:113 Beat name: WUWVC5ASGRD05.prod.wudip.com
2021-05-26T14:08:56.081-0400 WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-05-26T14:08:56.081-0400 INFO instance/beat.go:450 filebeat start running.
2021-05-26T14:08:56.081-0400 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-05-26T14:08:56.090-0400 INFO memlog/store.go:119 Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=913049512
2021-05-26T14:08:56.277-0400 INFO memlog/store.go:124 Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=913061217
2021-05-26T14:08:56.278-0400 WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-05-26T14:08:56.279-0400 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 271
2021-05-26T14:08:56.279-0400 INFO [crawler] beater/crawler.go:71 Loading Inputs: 18
2021-05-26T14:08:56.279-0400 INFO log/input.go:157 Configured paths: [/apps/csbatchjobs/01/logs/CsBatchJobsApplication.log]
2021-05-26T14:08:56.279-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 12568952622989063342)
2021-05-26T14:08:56.280-0400 INFO log/input.go:157 Configured paths: [/apps/wudgtsrvs/01/logs/cs.log]
2021-05-26T14:08:56.280-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 15999947131149582069)
2021-05-26T14:08:56.292-0400 INFO log/harvester.go:299 Harvester started for file: /apps/csbatchjobs/01/logs/CsBatchJobsApplication.log
2021-05-26T14:08:56.301-0400 INFO log/input.go:157 Configured paths: [/apps/wudgtsrvs/01/logs/logger.log]
2021-05-26T14:08:56.301-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 13264931403400537947)
2021-05-26T14:08:56.306-0400 INFO log/harvester.go:299 Harvester started for file: /apps/wudgtsrvs/01/logs/cs.log
2021-05-26T14:08:56.328-0400 INFO log/harvester.go:299 Harvester started for file: /apps/wudgtsrvs/01/logs/logger.log
2021-05-26T14:08:56.335-0400 INFO log/input.go:157 Configured paths: [/apps/wudgtsrvs/01/logs/CryptoUtil/cryptoUtil.log]
2021-05-26T14:08:56.335-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 341929948847540286)
2021-05-26T14:08:56.340-0400 INFO log/harvester.go:299 Harvester started for file: /apps/wudgtsrvs/01/logs/CryptoUtil/cryptoUtil.log
2021-05-26T14:08:56.429-0400 INFO log/input.go:157 Configured paths: [/apps/wudgtsrvs/02/logs/cs.log]
2021-05-26T14:08:56.430-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 5861920835871012163)
2021-05-26T14:08:56.440-0400 INFO log/harvester.go:299 Harvester started for file: /apps/wudgtsrvs/02/logs/cs.log
2021-05-26T14:08:56.450-0400 INFO log/input.go:157 Configured paths: [/apps/wudgtsrvs/02/logs/logger.log]
2021-05-26T14:08:56.450-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 12002253451971873368)
2021-05-26T14:08:56.461-0400 INFO log/harvester.go:299 Harvester started for file: /apps/wudgtsrvs/02/logs/logger.log
2021-05-26T14:08:56.466-0400 INFO log/input.go:157 Configured paths: [/apps/wudgtsrvs/02/logs/CryptoUtil/cryptoUtil.log]
2021-05-26T14:08:56.466-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 2665132126572961064)
2021-05-26T14:08:56.482-0400 INFO log/harvester.go:299 Harvester started for file: /apps/wudgtsrvs/02/logs/CryptoUtil/cryptoUtil.log
2021-05-26T14:08:56.508-0400 INFO log/input.go:157 Configured paths: [/apps/mobiliser5.1.0/01/money/logs/mobiliser.log /apps/mobiliser5.1.0/02/money/logs/mobiliser.log]
2021-05-26T14:08:56.508-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 9129910117981351875)
2021-05-26T14:08:56.518-0400 INFO log/input.go:157 Configured paths: [/apps/mobiliser5.1.0/01/money/logs/cs.log /apps/mobiliser5.1.0/02/money/logs/cs.log]
2021-05-26T14:08:56.518-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 18015971531683765891)
2021-05-26T14:08:56.528-0400 INFO log/input.go:157 Configured paths: [/apps/mobiliser5.1.0/01/money/logs/felix.out /apps/mobiliser5.1.0/02/money/logs/felix.out]
2021-05-26T14:08:56.528-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 4472731012317808238)
2021-05-26T14:08:56.539-0400 INFO log/input.go:157 Configured paths: [/apps/ps/01/felix-framework-4.4.0/logger.log /apps/ps/02/felix-framework-4.4.0/logger.log]
2021-05-26T14:08:56.539-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 2701953225666303299)
2021-05-26T14:08:56.544-0400 INFO log/input.go:157 Configured paths: [/apps/videokyc/logs/videokyc.log*]
2021-05-26T14:08:56.544-0400 INFO [crawler] beater/crawler.go:141 Starting input (ID: 7303334657564616302)
2021-05-26T14:08:56.545-0400 INFO beater/crawler.go:148 Stopping Crawler
2021-05-26T14:08:56.545-0400 INFO beater/crawler.go:158 Stopping 12 inputs
2021-05-26T14:08:56.545-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 9129910117981351875
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 5861920835871012163
2021-05-26T14:08:56.545-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 12568952622989063342
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 341929948847540286
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 2701953225666303299
2021-05-26T14:08:56.545-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 7303334657564616302
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 4472731012317808238
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 18015971531683765891
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 12002253451971873368
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 2665132126572961064
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 15999947131149582069
2021-05-26T14:08:56.546-0400 INFO [crawler] beater/crawler.go:163 Stopping input: 13264931403400537947
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.547-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.546-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:56.548-0400 INFO log/input.go:488 Scan aborted because input stopped.
2021-05-26T14:08:56.549-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:59.073-0400 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
2021-05-26T14:08:59.074-0400 INFO input/input.go:136 input ticker stopped
2021-05-26T14:08:59.074-0400 INFO beater/crawler.go:178 Crawler stopped
2021-05-26T14:08:59.074-0400 INFO [registrar] registrar/registrar.go:132 Stopping Registrar
2021-05-26T14:08:59.074-0400 INFO [registrar] registrar/registrar.go:166 Ending Registrar
2021-05-26T14:08:59.083-0400 INFO [registrar] registrar/registrar.go:137 Registrar stopped
2021-05-26T14:08:59.097-0400 INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":150}},"total":{"ticks":1060,"time":{"ms":1065},"value":1060},"user":{"ticks":910,"time":{"ms":915}}},"handles":{"limit":{"hard":131072,"soft":131072},"open":8},"info":{"ephemeral_id":"61e99822-12e0-423a-a463-6b9b9616eb97","uptime":{"ms":3103}},"memstats":{"gc_next":19985648,"memory_alloc":12525872,"memory_total":112418456,"rss":73617408},"runtime":{"goroutines":20}},"filebeat":{"events":{"active":37,"added":82,"done":45},"harvester":{"closed":7,"open_files":0,"running":0,"started":7}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":16,"filtered":66,"published":16,"total":82}}},"registrar":{"states":{"cleanup":6,"current":269,"update":45},"writes":{"success":45,"total":45}},"system":{"cpu":{"cores":40},"load":{"1":0.33,"15":0.41,"5":0.34,"norm":{"1":0.0083,"15":0.0102,"5":0.0085}}}}}}
2021-05-26T14:08:59.097-0400 INFO [monitoring] log/log.go:154 Uptime: 3.103648232s
2021-05-26T14:08:59.097-0400 INFO [monitoring] log/log.go:131 Stopping metrics logging.
2021-05-26T14:08:59.097-0400 INFO instance/beat.go:456 filebeat stopped.
2021-05-26T14:08:59.098-0400 ERROR instance/beat.go:951 Exiting: Failed to start crawler: starting input failed: Error while initializing input: Can only start an input when all related states are finished: {Id: native::15730558-64772, Finished: false, Fileinfo: &{cs.log 17453 436 {548755813 63757649317 0x607ff60} {64772 15730558 1 33204 3005 3005 0 0 17453 4096 40 {1622052519 404792930} {1622052517 548755813} {1622052517 548755813} [0 0 0]}}, Source: /apps/wudgtsrvs/01/logs/cs.log, Offset: 0, Timestamp: 2021-05-26 14:08:56.311368303 -0400 EDT m=+0.336775469, TTL: -1ns, Type: log, Meta: map[], FileStateOS: 15730558-64772}