Start request repeated too quickly for filebeat.service

Elastic Stack Beats
1

Hello All,

I have several servers set up with Filebeat since few months , nothing changed in terms of config for any.

Over the past few days I am observing 2 of these servers consistently failing to keep Filebeat service alive even if trying to start via systemctl ( I have given sufficient time in between commands to make sure all threads close ).

OS - RHEL 7.9
filebeat version 7.6.0 (amd64), libbeat 7.6.0 [6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c built 2020-02-05 23:06:45 +0000 UTC]

This might have been discussed as part of a different topic, but I wanted a fresh look at it.
Curious to know a way out of this issue.

$ filebeat  test config --path.config /etc/filebeat
Config OK
$ filebeat  test output --path.config /etc/filebeat
logstash: x.x.x.x:xxxxx...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: x.x.x.x
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
logstash: x.x.x.x:xxxxx...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: x.x.x.x
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK
logstash: x.x.x.x:xxxxx...
  connection...
    parse host... OK
    dns lookup... OK
    addresses: x.x.x.x
    dial up... OK
  TLS... WARN secure connection disabled
  talk to server... OK

The issue starts here -

$ systemctl status filebeat
* filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2022-05-18 11:03:21 AEST; 4min 39s ago
     Docs: https://www.elastic.co/products/beats/filebeat
  Process: 11567 ExecStart=/usr/share/filebeat/bin/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 11567 (code=exited, status=1/FAILURE)

May 18 11:03:21 xxxx.yyyy systemd[1]: Unit filebeat.service entered failed state.
May 18 11:03:21 xxxx.yyyy systemd[1]: filebeat.service failed.
May 18 11:03:21 xxxx.yyyy systemd[1]: filebeat.service holdoff time over, scheduling restart.
May 18 11:03:21 xxxx.yyyy systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:03:21 xxxx.yyyy systemd[1]: start request repeated too quickly for filebeat.service
May 18 11:03:21 xxxx.yyyy systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:03:21 xxxx.yyyy systemd[1]: Unit filebeat.service entered failed state.
May 18 11:03:21 xxxx.yyyy systemd[1]: filebeat.service failed.


$ systemctl start filebeat

$ systemctl status filebeat
* filebeat.service - Filebeat sends log files to Logstash or directly to Elasticsearch.
   Loaded: loaded (/usr/lib/systemd/system/filebeat.service; enabled; vendor preset: disabled)
   Active: failed (Result: start-limit) since Wed 2022-05-18 11:08:16 AEST; 25s ago
     Docs: https://www.elastic.co/products/beats/filebeat
  Process: 16593 ExecStart=/usr/share/filebeat/bin/filebeat $BEAT_LOG_OPTS $BEAT_CONFIG_OPTS $BEAT_PATH_OPTS (code=exited, status=1/FAILURE)
 Main PID: 16593 (code=exited, status=1/FAILURE)

May 18 11:08:16 xxxx.yyyy systemd[1]: Unit filebeat.service entered failed state.
May 18 11:08:16 xxxx.yyyy systemd[1]: filebeat.service failed.
May 18 11:08:16 xxxx.yyyy systemd[1]: filebeat.service holdoff time over, scheduling restart.
May 18 11:08:16 xxxx.yyyy systemd[1]: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:08:16 xxxx.yyyy systemd[1]: start request repeated too quickly for filebeat.service
May 18 11:08:16 xxxx.yyyy systemd[1]: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:08:16 xxxx.yyyy systemd[1]: Unit filebeat.service entered failed state.
May 18 11:08:16 xxxx.yyyy systemd[1]: filebeat.service failed.
2

Welcome to our community! :smiley:
As a tip, format your code/logs/config using the </> button, or markdown style back ticks. It helps to make things easy to read which helps us help you.

What do your actual Filebeat logs show?

3

This was my 1st post and realised my mistake , apologies indeed.
Portion of log attached herewith.

May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.129+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T084848
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.130+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T093827
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.129+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T084848
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.130+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T093827
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.131+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T050147
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.131+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T050147
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.133+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T065836
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.133+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T091605
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.133+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T065836
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.133+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/noapp.log.D20220518.T091605
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.145+1000#011INFO#011pipeline/output.go:95#011Connecting to backoff(async xxxx)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.145+1000#011INFO#011pipeline/output.go:95#011Connecting to backoff(async xxxx)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.145+1000#011INFO#011pipeline/output.go:95#011Connecting to backoff(async xxxx)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.145+1000#011INFO#011pipeline/output.go:95#011Connecting to backoff(async xxxx)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.145+1000#011INFO#011pipeline/output.go:95#011Connecting to backoff(async xxxx)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.145+1000#011INFO#011pipeline/output.go:95#011Connecting to backoff(async xxxx)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.149+1000#011INFO#011pipeline/output.go:105#011Connection to backoff(async xxxx established)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.150+1000#011INFO#011pipeline/output.go:105#011Connection to backoff((async xxxx established)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.149+1000#011INFO#011pipeline/output.go:105#011Connection to backoff(async xxxx established)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.150+1000#011INFO#011pipeline/output.go:105#011Connection to backoff(async xxxx established)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.150+1000#011INFO#011pipeline/output.go:105#011Connection to backoff(async xxxx established)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.150+1000#011INFO#011pipeline/output.go:105#011Connection to backoff(async xxxx established)
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.164+1000#011INFO#011log/input.go:152#011Configured paths: [/opt/ibm/B2Bi/install/logs/jetty.log.*]
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.164+1000#011INFO#011input/input.go:114#011Starting input of type: log; ID: 3967677331675005990
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.164+1000#011INFO#011log/input.go:152#011Configured paths: [/opt/ibm/B2Bi/install/logs/jetty.log.*]
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.164+1000#011INFO#011input/input.go:114#011Starting input of type: log; ID: 3967677331675005990
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.169+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/jetty.log.D20220509.T153552
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.169+1000#011INFO#011log/harvester.go:297#011Harvester started for file: /opt/ibm/B2Bi/install/logs/jetty.log.D20220509.T153552
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.170+1000#011INFO#011log/input.go:152#011Configured paths: [/opt/ibm/B2Bi/install/logs/wf.log.*]
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.170+1000#011INFO#011input/input.go:114#011Starting input of type: log; ID: 14382257620082020189
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.170+1000#011INFO#011log/input.go:152#011Configured paths: [/opt/ibm/B2Bi/install/logs/wf.log.*]
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.170+1000#011INFO#011input/input.go:114#011Starting input of type: log; ID: 14382257620082020189
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011crawler/crawler.go:139#011Stopping Crawler
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011crawler/crawler.go:149#011Stopping 6 inputs
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 3967677331675005990
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 4471271906142236015
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011crawler/crawler.go:139#011Stopping Crawler
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011crawler/crawler.go:149#011Stopping 6 inputs
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 3967677331675005990
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 4471271906142236015
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 743785949444259268
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 4408760385737834315
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 3234903351282687959
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011ERROR#011log/input.go:501#011Harvester could not be started on new file: /opt/ibm/B2Bi/install/logs/wf.log.D20220516.T000000, Err: registry already stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/input.go:470#011Scan aborted because input stopped.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/input.go:722#011input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011ERROR#011log/input.go:250#011File cleanup state update error: input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/input.go:722#011input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011ERROR#011log/input.go:250#011File cleanup state update error: input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 743785949444259268
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 4408760385737834315
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011input/input.go:167#011Stopping Input: 3234903351282687959
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.171+1000#011ERROR#011log/input.go:501#011Harvester could not be started on new file: /opt/ibm/B2Bi/install/logs/wf.log.D20220516.T000000, Err: registry already stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/input.go:470#011Scan aborted because input stopped.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/input.go:722#011input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011ERROR#011log/input.go:250#011File cleanup state update error: input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/input.go:722#011input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011ERROR#011log/input.go:250#011File cleanup state update error: input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011input/input.go:167#011Stopping Input: 14382257620082020189
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011input/input.go:149#011input ticker stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011input/input.go:167#011Stopping Input: 14382257620082020189
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/harvester.go:320#011Reader was closed: /opt/ibm/B2Bi/install/logs/ops.log.D20220514.T000000. Closing.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011crawler/crawler.go:165#011Crawler stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011registrar/registrar.go:367#011Stopping Registrar
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.179+1000#011INFO#011registrar/registrar.go:293#011Ending Registrar
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011[monitoring]#011log/log.go:153#011Total non-zero metrics#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":113}},"total":{"ticks":360,"time":{"ms":368},"value":360},"user":{"ticks":250,"time":{"ms":255}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"c03c549d-5da4-4a3b-acea-9fbbbac2604c","uptime":{"ms":279}},"memstats":{"gc_next":32001152,"memory_alloc":26308208,"memory_total":58365624,"rss":51654656},"runtime":{"goroutines":35}},"filebeat":{"events":{"active":3531,"added":3617,"done":86},"harvester":{"closed":44,"files":{"16ea31c7-5ea6-4d99-a771-340161388a6b":{"last_event_published_time":"","last_event_timestamp":"","name":"/opt/ibm/B2Bi/install/logs/wf.log.D20220516.T000000","size":26972708,"start_time":"2022-05-18T11:08:16.171Z"}},"open_files":1,"running":0,"started":44}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":2048,"batches":1,"total":2048},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":3431,"filtered":186,"published":3431,"retry":6144,"total":3617}}},"registrar":{"states":{"current":89,"update":86},"writes":{"success":87,"total":87}},"system":{"cpu":{"cores":8},"load":{"1":0.33,"15":0.42,"5":0.37,"norm":{"1":0.0413,"15":0.0525,"5":0.0463}}}}}}
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011[monitoring]#011log/log.go:154#011Uptime: 281.080493ms
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011[monitoring]#011log/log.go:131#011Stopping metrics logging.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011instance/beat.go:445#011filebeat stopped.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011ERROR#011instance/beat.go:933#011Exiting: Error while initializing input: Can only start an input when all related states are finished: {Id:61553-64768 Finished:false Fileinfo:0xc00063b6c0 Source:/opt/ibm/B2Bi/install/logs/ui.log.D20220509.T153523 Offset:0 Timestamp:2022-05-18 11:08:15.962761598 +1000 AEST m=+0.062903729 TTL:-1ns Type:log Meta:map[] FileStateOS:61553-64768}
May 18 11:08:16 xxxx filebeat: Exiting: Error while initializing input: Can only start an input when all related states are finished: {Id:61553-64768 Finished:false Fileinfo:0xc00063b6c0 Source:/opt/ibm/B2Bi/install/logs/ui.log.D20220509.T153523 Offset:0 Timestamp:2022-05-18 11:08:15.962761598 +1000 AEST m=+0.062903729 TTL:-1ns Type:log Meta:map[] FileStateOS:61553-64768}
May 18 11:08:16 xxxx systemd: filebeat.service: main process exited, code=exited, status=1/FAILURE
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011log/harvester.go:320#011Reader was closed: /opt/ibm/B2Bi/install/logs/ops.log.D20220514.T000000. Closing.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011harvester/forwarder.go:52#011Input outlet closed
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011crawler/crawler.go:165#011Crawler stopped
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.172+1000#011INFO#011registrar/registrar.go:367#011Stopping Registrar
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.179+1000#011INFO#011registrar/registrar.go:293#011Ending Registrar
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011[monitoring]#011log/log.go:153#011Total non-zero metrics#011{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":110,"time":{"ms":113}},"total":{"ticks":360,"time":{"ms":368},"value":360},"user":{"ticks":250,"time":{"ms":255}}},"handles":{"limit":{"hard":4096,"soft":1024},"open":11},"info":{"ephemeral_id":"c03c549d-5da4-4a3b-acea-9fbbbac2604c","uptime":{"ms":279}},"memstats":{"gc_next":32001152,"memory_alloc":26308208,"memory_total":58365624,"rss":51654656},"runtime":{"goroutines":35}},"filebeat":{"events":{"active":3531,"added":3617,"done":86},"harvester":{"closed":44,"files":{"16ea31c7-5ea6-4d99-a771-340161388a6b":{"last_event_published_time":"","last_event_timestamp":"","name":"/opt/ibm/B2Bi/install/logs/wf.log.D20220516.T000000","size":26972708,"start_time":"2022-05-18T11:08:16.171Z"}},"open_files":1,"running":0,"started":44}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":2048,"batches":1,"total":2048},"type":"logstash"},"pipeline":{"clients":0,"events":{"active":3431,"filtered":186,"published":3431,"retry":6144,"total":3617}}},"registrar":{"states":{"current":89,"update":86},"writes":{"success":87,"total":87}},"system":{"cpu":{"cores":8},"load":{"1":0.33,"15":0.42,"5":0.37,"norm":{"1":0.0413,"15":0.0525,"5":0.0463}}}}}}
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011[monitoring]#011log/log.go:154#011Uptime: 281.080493ms
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011[monitoring]#011log/log.go:131#011Stopping metrics logging.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011INFO#011instance/beat.go:445#011filebeat stopped.
May 18 11:08:16 xxxx filebeat: 2022-05-18T11:08:16.184+1000#011ERROR#011instance/beat.go:933#011Exiting: Error while initializing input: Can only start an input when all related states are finished: {Id:61553-64768 Finished:false Fileinfo:0xc00063b6c0 Source:/opt/ibm/B2Bi/install/logs/ui.log.D20220509.T153523 Offset:0 Timestamp:2022-05-18 11:08:15.962761598 +1000 AEST m=+0.062903729 TTL:-1ns Type:log Meta:map[] FileStateOS:61553-64768}
May 18 11:08:16 xxxx filebeat: Exiting: Error while initializing input: Can only start an input when all related states are finished: {Id:61553-64768 Finished:false Fileinfo:0xc00063b6c0 Source:/opt/ibm/B2Bi/install/logs/ui.log.D20220509.T153523 Offset:0 Timestamp:2022-05-18 11:08:15.962761598 +1000 AEST m=+0.062903729 TTL:-1ns Type:log Meta:map[] FileStateOS:61553-64768}
May 18 11:08:16 xxxx systemd: filebeat.service: main process exited, code=exited, status=1/FAILURE
May 18 11:08:16 xxxx systemd: Unit filebeat.service entered failed state.
May 18 11:08:16 xxxx systemd: filebeat.service failed.
May 18 11:08:16 xxxx systemd: Unit filebeat.service entered failed state.
May 18 11:08:16 xxxx systemd: filebeat.service failed.
May 18 11:08:16 xxxx systemd: filebeat.service holdoff time over, scheduling restart.
May 18 11:08:16 xxxx systemd: filebeat.service holdoff time over, scheduling restart.
May 18 11:08:16 xxxx systemd: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:08:16 xxxx systemd: Stopped Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:08:16 xxxx systemd: start request repeated too quickly for filebeat.service
May 18 11:08:16 xxxx systemd: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:08:16 xxxx systemd: start request repeated too quickly for filebeat.service
May 18 11:08:16 xxxx systemd: Failed to start Filebeat sends log files to Logstash or directly to Elasticsearch..
May 18 11:08:16 xxxx systemd: Unit filebeat.service entered failed state.
May 18 11:08:16 xxxx systemd: filebeat.service failed.
May 18 11:08:16 xxxx systemd: Unit filebeat.service entered failed state.
May 18 11:08:16 xxxx systemd: filebeat.service failed.
4

The reported issue resolved itself the next day when I tried.
Likely to pop up in another server anytime soon.

Appreciate if anyone can provide any further insights that I might investigate.

5

This issue appears on few servers repeatedly through out every day.
Manual start also fails to stay alive beyond a few minutes.

Anyone with more knowledge on these if can help would be appreciated.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.