Unable to start and stop filebeat

Hello team,
I am unable to start and stop filebeat . getting error like failed to stop filebeat.

Can you please help on this.

PFB filebeat logs:

2021-05-26T14:08:56.070-0400	INFO	instance/beat.go:640	Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2021-05-26T14:08:56.071-0400	INFO	instance/beat.go:648	Beat ID: ce29b12d-0ea9-4044-9312-e3c9791201b8
2021-05-26T14:08:56.074-0400	INFO	[seccomp]	seccomp/seccomp.go:101	Syscall filter could not be installed because the kernel does not support seccomp
2021-05-26T14:08:56.074-0400	INFO	[beat]	instance/beat.go:976	Beat info	{"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "ce29b12d-0ea9-4044-9312-e3c9791201b8"}}}
2021-05-26T14:08:56.074-0400	INFO	[beat]	instance/beat.go:985	Build info	{"system_info": {"build": {"commit": "2ab907f5ccecf9fd82fe37105082e89fd871f684", "libbeat": "7.9.2", "time": "2020-09-22T23:19:45.000Z", "version": "7.9.2"}}}
2021-05-26T14:08:56.074-0400	INFO	[beat]	instance/beat.go:988	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":40,"version":"go1.14.7"}}}
2021-05-26T14:08:56.075-0400	INFO	[beat]	instance/beat.go:992	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-04-07T23:30:21-04:00","containerized":false,"name":"WUWVC5ASGRD05.prod.wudip.com","ip":["127.0.0.1/8","10.45.16.205/22"],"kernel_version":"2.6.32-754.36.1.el6.x86_64","mac":["00:17:a4:77:38:30","00:17:a4:77:38:30","00:17:a4:77:38:2e","00:17:a4:77:38:2f","00:17:a4:77:38:30"],"os":{"family":"redhat","platform":"redhat","name":"Red","version":"6.10 (Santiago)","major":6,"minor":10,"patch":0,"codename":"Santiago"},"timezone":"EDT","timezone_offset_sec":-14400,"id":"b65197304d2cb376f829ef3f00000011"}}}
2021-05-26T14:08:56.076-0400	INFO	instance/beat.go:299	Setup Beat: filebeat; Version: 7.9.2
2021-05-26T14:08:56.079-0400	INFO	[publisher]	pipeline/module.go:113	Beat name: WUWVC5ASGRD05.prod.wudip.com
2021-05-26T14:08:56.081-0400	WARN	beater/filebeat.go:178	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-05-26T14:08:56.081-0400	INFO	instance/beat.go:450	filebeat start running.
2021-05-26T14:08:56.081-0400	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2021-05-26T14:08:56.090-0400	INFO	memlog/store.go:119	Loading data file of '/var/lib/filebeat/registry/filebeat' succeeded. Active transaction id=913049512
2021-05-26T14:08:56.277-0400	INFO	memlog/store.go:124	Finished loading transaction log file for '/var/lib/filebeat/registry/filebeat'. Active transaction id=913061217
2021-05-26T14:08:56.278-0400	WARN	beater/filebeat.go:381	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-05-26T14:08:56.279-0400	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 271
2021-05-26T14:08:56.279-0400	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 18
2021-05-26T14:08:56.279-0400	INFO	log/input.go:157	Configured paths: [/apps/csbatchjobs/01/logs/CsBatchJobsApplication.log]
2021-05-26T14:08:56.279-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 12568952622989063342)
2021-05-26T14:08:56.280-0400	INFO	log/input.go:157	Configured paths: [/apps/wudgtsrvs/01/logs/cs.log]
2021-05-26T14:08:56.280-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 15999947131149582069)
2021-05-26T14:08:56.292-0400	INFO	log/harvester.go:299	Harvester started for file: /apps/csbatchjobs/01/logs/CsBatchJobsApplication.log
2021-05-26T14:08:56.301-0400	INFO	log/input.go:157	Configured paths: [/apps/wudgtsrvs/01/logs/logger.log]
2021-05-26T14:08:56.301-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 13264931403400537947)
2021-05-26T14:08:56.306-0400	INFO	log/harvester.go:299	Harvester started for file: /apps/wudgtsrvs/01/logs/cs.log
2021-05-26T14:08:56.328-0400	INFO	log/harvester.go:299	Harvester started for file: /apps/wudgtsrvs/01/logs/logger.log
2021-05-26T14:08:56.335-0400	INFO	log/input.go:157	Configured paths: [/apps/wudgtsrvs/01/logs/CryptoUtil/cryptoUtil.log]
2021-05-26T14:08:56.335-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 341929948847540286)
2021-05-26T14:08:56.340-0400	INFO	log/harvester.go:299	Harvester started for file: /apps/wudgtsrvs/01/logs/CryptoUtil/cryptoUtil.log
2021-05-26T14:08:56.429-0400	INFO	log/input.go:157	Configured paths: [/apps/wudgtsrvs/02/logs/cs.log]
2021-05-26T14:08:56.430-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 5861920835871012163)
2021-05-26T14:08:56.440-0400	INFO	log/harvester.go:299	Harvester started for file: /apps/wudgtsrvs/02/logs/cs.log
2021-05-26T14:08:56.450-0400	INFO	log/input.go:157	Configured paths: [/apps/wudgtsrvs/02/logs/logger.log]
2021-05-26T14:08:56.450-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 12002253451971873368)
2021-05-26T14:08:56.461-0400	INFO	log/harvester.go:299	Harvester started for file: /apps/wudgtsrvs/02/logs/logger.log
2021-05-26T14:08:56.466-0400	INFO	log/input.go:157	Configured paths: [/apps/wudgtsrvs/02/logs/CryptoUtil/cryptoUtil.log]
2021-05-26T14:08:56.466-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 2665132126572961064)
2021-05-26T14:08:56.482-0400	INFO	log/harvester.go:299	Harvester started for file: /apps/wudgtsrvs/02/logs/CryptoUtil/cryptoUtil.log
2021-05-26T14:08:56.508-0400	INFO	log/input.go:157	Configured paths: [/apps/mobiliser5.1.0/01/money/logs/mobiliser.log /apps/mobiliser5.1.0/02/money/logs/mobiliser.log]
2021-05-26T14:08:56.508-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 9129910117981351875)
2021-05-26T14:08:56.518-0400	INFO	log/input.go:157	Configured paths: [/apps/mobiliser5.1.0/01/money/logs/cs.log /apps/mobiliser5.1.0/02/money/logs/cs.log]
2021-05-26T14:08:56.518-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 18015971531683765891)
2021-05-26T14:08:56.528-0400	INFO	log/input.go:157	Configured paths: [/apps/mobiliser5.1.0/01/money/logs/felix.out /apps/mobiliser5.1.0/02/money/logs/felix.out]
2021-05-26T14:08:56.528-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 4472731012317808238)
2021-05-26T14:08:56.539-0400	INFO	log/input.go:157	Configured paths: [/apps/ps/01/felix-framework-4.4.0/logger.log /apps/ps/02/felix-framework-4.4.0/logger.log]
2021-05-26T14:08:56.539-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 2701953225666303299)
2021-05-26T14:08:56.544-0400	INFO	log/input.go:157	Configured paths: [/apps/videokyc/logs/videokyc.log*]
2021-05-26T14:08:56.544-0400	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 7303334657564616302)
2021-05-26T14:08:56.545-0400	INFO	beater/crawler.go:148	Stopping Crawler
2021-05-26T14:08:56.545-0400	INFO	beater/crawler.go:158	Stopping 12 inputs
2021-05-26T14:08:56.545-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 9129910117981351875
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 5861920835871012163
2021-05-26T14:08:56.545-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 12568952622989063342
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 341929948847540286
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 2701953225666303299
2021-05-26T14:08:56.545-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 7303334657564616302
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 4472731012317808238
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 18015971531683765891
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 12002253451971873368
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 2665132126572961064
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 15999947131149582069
2021-05-26T14:08:56.546-0400	INFO	[crawler]	beater/crawler.go:163	Stopping input: 13264931403400537947
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.547-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.546-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:56.548-0400	INFO	log/input.go:488	Scan aborted because input stopped.
2021-05-26T14:08:56.549-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:59.073-0400	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2021-05-26T14:08:59.074-0400	INFO	input/input.go:136	input ticker stopped
2021-05-26T14:08:59.074-0400	INFO	beater/crawler.go:178	Crawler stopped
2021-05-26T14:08:59.074-0400	INFO	[registrar]	registrar/registrar.go:132	Stopping Registrar
2021-05-26T14:08:59.074-0400	INFO	[registrar]	registrar/registrar.go:166	Ending Registrar
2021-05-26T14:08:59.083-0400	INFO	[registrar]	registrar/registrar.go:137	Registrar stopped
2021-05-26T14:08:59.097-0400	INFO	[monitoring]	log/log.go:153	Total non-zero metrics	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":150,"time":{"ms":150}},"total":{"ticks":1060,"time":{"ms":1065},"value":1060},"user":{"ticks":910,"time":{"ms":915}}},"handles":{"limit":{"hard":131072,"soft":131072},"open":8},"info":{"ephemeral_id":"61e99822-12e0-423a-a463-6b9b9616eb97","uptime":{"ms":3103}},"memstats":{"gc_next":19985648,"memory_alloc":12525872,"memory_total":112418456,"rss":73617408},"runtime":{"goroutines":20}},"filebeat":{"events":{"active":37,"added":82,"done":45},"harvester":{"closed":7,"open_files":0,"running":0,"started":7}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":16,"filtered":66,"published":16,"total":82}}},"registrar":{"states":{"cleanup":6,"current":269,"update":45},"writes":{"success":45,"total":45}},"system":{"cpu":{"cores":40},"load":{"1":0.33,"15":0.41,"5":0.34,"norm":{"1":0.0083,"15":0.0102,"5":0.0085}}}}}}
2021-05-26T14:08:59.097-0400	INFO	[monitoring]	log/log.go:154	Uptime: 3.103648232s
2021-05-26T14:08:59.097-0400	INFO	[monitoring]	log/log.go:131	Stopping metrics logging.
2021-05-26T14:08:59.097-0400	INFO	instance/beat.go:456	filebeat stopped.
2021-05-26T14:08:59.098-0400	ERROR	instance/beat.go:951	Exiting: Failed to start crawler: starting input failed: Error while initializing input: Can only start an input when all related states are finished: {Id: native::15730558-64772, Finished: false, Fileinfo: &{cs.log 17453 436 {548755813 63757649317 0x607ff60} {64772 15730558 1 33204 3005 3005 0 0 17453 4096 40 {1622052519 404792930} {1622052517 548755813} {1622052517 548755813} [0 0 0]}}, Source: /apps/wudgtsrvs/01/logs/cs.log, Offset: 0, Timestamp: 2021-05-26 14:08:56.311368303 -0400 EDT m=+0.336775469, TTL: -1ns, Type: log, Meta: map[], FileStateOS: 15730558-64772}

Did you try to clean the registry file? It looks like something is messed up.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.