I have installed Filebeat 5.2.0 on windows 2012 R2 server. The issue I have is the filebeat service stops at a particular time on most of our servers. The Log file does not have any error but just info. I have set the service startup type as automatic also have set Recovery after failure, but still for some reason service stops everyday and it doesn't restart. I have to manually start the service which is not desirable. Any help please?
Here is the Filebeat Config
filebeat.prospectors:
-
input_type: log
paths:
- D:\iislogs***.log
tail_files: true
- D:\iislogs***.log
fields:
environment: ${Server}
platform: ${Platform}
output.kafka:
hosts: ["dummykafka:9092","dummykafka:9092"]
topic: 'kafkatopic.IISLog'
partition.round_robin:
reachable_only: false
required_acks: 1
compression: gzip
max_message_bytes: 1000000
Here is the Error Log from one of the server
2018-04-12T02:37:01-05:00 INFO Home path: [D:\Source\Filebeat] Config path: [D:\Source\Filebeat] Data path: [D:\Source\Filebeat\data] Logs path: [D:\Source\Filebeat\logs]
2018-04-12T02:37:01-05:00 INFO Setup Beat: filebeat; Version: 5.2.0
2018-04-12T02:37:01-05:00 INFO Activated kafka as output plugin.
2018-04-12T02:37:01-05:00 INFO Publisher name: mgowiauaq2
2018-04-12T02:37:01-05:00 INFO Flush Interval set to: 1s
2018-04-12T02:37:01-05:00 INFO Max Bulk Size set to: 2048
2018-04-12T02:37:01-05:00 INFO filebeat start running.
2018-04-12T02:37:01-05:00 INFO Registry file set to: D:\Source\Filebeat\data\registry
2018-04-12T02:37:01-05:00 INFO Loading registrar data from D:\Source\Filebeat\data\registry
2018-04-12T02:37:01-05:00 INFO States Loaded from registrar: 911
2018-04-12T02:37:01-05:00 INFO Loading Prospectors: 1
2018-04-12T02:37:01-05:00 INFO Start sending events to output
2018-04-12T02:37:01-05:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-04-12T02:37:01-05:00 INFO Starting Registrar
2018-04-12T02:37:01-05:00 INFO Prospector with previous states loaded: 911
2018-04-12T02:37:01-05:00 INFO Loading Prospectors completed. Number of prospectors: 1
2018-04-12T02:37:01-05:00 INFO All prospectors are initialised and running with 911 states to persist
2018-04-12T02:37:01-05:00 INFO Starting prospector of type: log
2018-04-12T02:37:11-05:00 INFO Harvester started for file: D:\iislogs\abc\W3SVC10851\u_ex180411.log
2018-04-12T02:37:11-05:00 INFO Harvester started for file: D:\iislogs\xyz\W3SVC10826\u_ex180411.log
2018-04-12T02:37:11-05:00 INFO Harvester started for file: D:\iislogs\qwe\W3SVC10963\u_ex180411.log
2018-04-12T02:37:11-05:00 INFO Harvester started for file: D:\iislogs\rty\W3SVC10949\u_ex180411.log
2018-04-12T23:57:33-05:00 INFO No non-zero metrics in the last 30s
2018-04-12T23:58:03-05:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=364 publish.events=364 libbeat.kafka.call_count.PublishEvents=2 libbeat.kafka.published_and_acked_events=364 registrar.states.update=364 registrar.writes=2
2018-04-13T00:05:33-05:00 INFO No non-zero metrics in the last 30s
2018-04-13T00:06:03-05:00 INFO Non-zero metrics in the last 30s: libbeat.publisher.published_events=364 registrar.states.update=364 libbeat.kafka.published_and_acked_events=364 publish.events=364 libbeat.kafka.call_count.PublishEvents=2 registrar.writes=2
2018-04-13T00:06:33-05:00 INFO No non-zero metrics in the last 30s
2018-04-13T00:07:03-05:00 INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=363 libbeat.kafka.call_count.PublishEvents=2 libbeat.publisher.published_events=363 libbeat.kafka.published_and_acked_events=363 publish.events=363
2018-04-13T00:07:13-05:00 WARN client/metadata fetching metadata for all topics from broker dummykafkabroker:9092
2018-04-13T00:07:13-05:00 WARN Connected to broker at dummykafkabroker:9092 (registered as #2)
2018-04-13T00:27:33-05:00 INFO No non-zero metrics in the last 30s
2018-04-13T00:28:03-05:00 INFO Non-zero metrics in the last 30s: libbeat.kafka.published_and_acked_events=364 libbeat.kafka.call_count.PublishEvents=2 publish.events=364 registrar.states.update=364 registrar.writes=2 libbeat.publisher.published_events=364
2018-04-13T00:28:33-05:00 INFO No non-zero metrics in the last 30s
2018-04-13T00:29:03-05:00 INFO Non-zero metrics in the last 30s: libbeat.kafka.call_count.PublishEvents=2 libbeat.kafka.published_and_acked_events=364 publish.events=364 libbeat.publisher.published_events=364 registrar.states.update=364 registrar.writes=2
2018-04-13T00:29:33-05:00 INFO No non-zero metrics in the last 30s
2018-04-13T00:30:01-05:00 INFO Stopping filebeat
2018-04-13T00:30:01-05:00 INFO Stopping Crawler
2018-04-13T00:30:01-05:00 INFO Stopping 1 prospectors
2018-04-13T00:30:01-05:00 INFO Stopping Prospector
2018-04-13T00:30:01-05:00 INFO Reader was closed: D:\iislogs\abc\W3SVC10945\u_ex180413.log. Closing.
2018-04-13T00:30:01-05:00 INFO Reader was closed: D:\iislogs\qwer\W3SVC10812\u_ex180413.log. Closing.
2018-04-13T00:30:01-05:00 INFO Reader was closed: D:\iislogs\zxcv\W3SVC10806\u_ex180413.log. Closing.
2018-04-13T00:30:01-05:00 INFO Prospector ticker stopped
2018-04-13T00:30:01-05:00 INFO Prospector channel stopped
2018-04-13T00:30:01-05:00 INFO Reader was closed: D:\iislogs\qwe\W3SVC10830\u_ex180413.log. Closing.
2018-04-13T00:30:01-05:00 INFO Reader was closed: D:\iislogs\ert\W3SVC10930\u_ex180413.log. Closing.
2018-04-13T00:30:01-05:00 INFO Reader was closed: D:\iislogs\sdf\W3SVC10987\u_ex180413.log. Closing.
2018-04-13T00:30:01-05:00 INFO Crawler stopped
2018-04-13T00:30:01-05:00 INFO Stopping spooler
2018-04-13T00:30:01-05:00 INFO Stopping Registrar
2018-04-13T00:30:01-05:00 INFO Ending Registrar
2018-04-13T00:30:01-05:00 INFO Total non-zero values: registrar.writes=2070 publish.events=892472 libbeat.publisher.published_events=890105 registar.states.current=911 registrar.states.cleanup=364 registrar.states.update=892472 libbeat.kafka.call_count.PublishEvents=2061 libbeat.kafka.published_and_acked_events=890105 filebeat.harvester.closed=546 filebeat.harvester.started=546
2018-04-13T00:30:01-05:00 INFO Uptime: 21h52m59.825078s
2018-04-13T00:30:01-05:00 INFO filebeat stopped.