Filebeat is not processing the logs which suppose to be sending to LogStash hosted in same Machine

Elastic Stack Beats
1

My filebeat configuration has been working fine. But it suddenly stopped working now.
I am confirming this by seeing the filebeat logs, there is no harvester logs.

But I used to see the harvester logs before.

When i restart the Filebeat service, by that time, it is starting the Harvest . But after few seconds, it is getting closed as per the configuration and the next Harvest is not starting at all.

This is the log i am getting every 30 sec.

2020-07-17T14:38:51.090+0530	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":46765,"time":46765},"total":{"ticks":97780,"time":97780,"value":97780},"user":{"ticks":51015,"time":51015}},"info":{"ephemeral_id":"e2dc09b1-fc09-48eb-ac1f-b7eb309b156e","uptime":{"ms":3240059}},"memstats":{"gc_next":12438416,"memory_alloc":9035784,"memory_total":2648809936,"rss":-49152}},"filebeat":{"harvester":{"open_files":5,"running":62}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":10,"events":{"active":4124}}},"registrar":{"states":{"current":1731}}}}}```
2

Please provide full output of filebeat log history aswell as your configuration files.

3

My File beat Config file looks below.

filebeat.prospectors:
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  paths:
    - "c:/Source/audit/**/*.txt"
  tail_files: true
  fields:
    item: Source_audit
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  paths:
    - "c:/Source/logs/**/*"
  tail_files: true
  fields:
    item: Source_logs
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  paths:
    - "c:/Source/connector/logs/**/*.log"
  tail_files: true
  fields:
    item: connector_logs
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  paths:
    - c:/Source/connector/status/*.status
  tail_files: true
  fields:
    item: connector_audit
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  ignore_older: 10m
  paths:
    - c:/Source/connector/CSV/**/*.log
  tail_files: true
  fields:
    item: connector_csv_logs
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  ignore_older: 10m
  paths:
    - c:/Source/connector/XML/**/*.log
  tail_files: true
  fields:
    item: connector_xml_logs
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  ignore_older: 10m
  paths:
    - c:/Source/connector/JSON/**/*.log
  tail_files: true
  fields:
    item: connector_json_logs
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  ignore_older: 10m
  paths:
    - c:/Source/connector/DBConnector/**/*.log
  tail_files: true
  fields:
    item: connector_dbc_logs
- type: log
  enabled: true
  close_renamed: true
  close_timeout: 1m
  close_removed: true
  paths:
    - c:/test/type1*/audit/*.txt
    - c:/test/type1*/audit/*.txt
    - c:/test/type1*/audit/*.txt
    - c:/test/type1/audit/*.txt
  tail_files: true
  fields:
    item: connector_audit
output.logstash:
  hosts: ["localhost:5045"]
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: true
  reload.period: 50s```
4

Due to log rotation policy i lost the history, here is the log after i restarted the Filebeat service.

2020-07-17T17:34:24.310+0530	INFO	instance/beat.go:468	Home path: [C:\Source\tools\logbeats] Config path: [C:\Source\tools\logbeats\] Data path: [C:\Source\tools\logbeats\data] Logs path: [C:\Source\tools\logbeats\logs]
2020-07-17T17:34:24.397+0530	INFO	instance/beat.go:475	Beat UUID: cfb2c93e-2578-445f-a590-eb4594854863
2020-07-17T17:34:24.397+0530	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.3
2020-07-17T17:34:24.398+0530	INFO	pipeline/module.go:76	Beat name: MachineName
2020-07-17T17:34:24.446+0530	INFO	instance/beat.go:301	filebeat start running.
2020-07-17T17:34:24.446+0530	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2020-07-17T17:34:24.461+0530	ERROR	service/service_windows.go:59	Error: The service process could not connect to the service controller.
2020-07-17T17:34:24.462+0530	INFO	registrar/registrar.go:108	Loading registrar data from C:\Source\tools\logbeats\data\registry
2020-07-17T17:34:24.486+0530	INFO	registrar/registrar.go:119	States Loaded from registrar: 1713
2020-07-17T17:34:24.486+0530	WARN	beater/filebeat.go:261	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-07-17T17:34:24.487+0530	INFO	crawler/crawler.go:48	Loading Prospectors: 10
2020-07-17T17:34:25.726+0530	INFO	log/prospector.go:111	Configured paths: [c:\source\audit\*.txt c:\source\audit\*\*.txt c:\source\audit\*\*\*.txt c:\source\audit\*\*\*\*.txt c:\source\audit\*\*\*\*\*.txt c:\source\audit\*\*\*\*\*\*.txt c:\source\audit\*\*\*\*\*\*\*.txt c:\source\audit\*\*\*\*\*\*\*\*.txt c:\source\audit\*\*\*\*\*\*\*\*\*.txt]
2020-07-17T17:34:29.718+0530	INFO	log/prospector.go:111	Configured paths: [c:\source\logs\* c:\source\logs\*\* c:\source\logs\*\*\* c:\source\logs\*\*\*\* c:\source\logs\*\*\*\*\* c:\source\logs\*\*\*\*\*\* c:\source\logs\*\*\*\*\*\*\* c:\source\logs\*\*\*\*\*\*\*\* c:\source\logs\*\*\*\*\*\*\*\*\*]
2020-07-17T17:34:29.775+0530	INFO	log/prospector.go:111	Configured paths: [c:\Source\connector\logs\*.log c:\Source\connector\logs\*\*.log c:\Source\connector\logs\*\*\*.log c:\Source\connector\logs\*\*\*\*.log c:\Source\connector\logs\*\*\*\*\*.log c:\Source\connector\logs\*\*\*\*\*\*.log c:\Source\connector\logs\*\*\*\*\*\*\*.log c:\Source\connector\logs\*\*\*\*\*\*\*\*.log c:\Source\connector\logs\*\*\*\*\*\*\*\*\*.log]
2020-07-17T17:34:29.995+0530	INFO	log/prospector.go:111	Configured paths: [c:\Source\connector\status\*.status]
2020-07-17T17:34:32.118+0530	INFO	log/prospector.go:111	Configured paths: [c:\Source\connector\CSV\*.log c:\Source\connector\CSV\*\*.log c:\Source\connector\CSV\*\*\*.log c:\Source\connector\CSV\*\*\*\*.log c:\Source\connector\CSV\*\*\*\*\*.log c:\Source\connector\CSV\*\*\*\*\*\*.log c:\Source\connector\CSV\*\*\*\*\*\*\*.log c:\Source\connector\CSV\*\*\*\*\*\*\*\*.log c:\Source\connector\CSV\*\*\*\*\*\*\*\*\*.log]
2020-07-17T17:34:37.659+0530	INFO	log/prospector.go:111	Configured paths: [c:\Source\connector\XML\*.log c:\Source\connector\XML\*\*.log c:\Source\connector\XML\*\*\*.log c:\Source\connector\XML\*\*\*\*.log c:\Source\connector\XML\*\*\*\*\*.log c:\Source\connector\XML\*\*\*\*\*\*.log c:\Source\connector\XML\*\*\*\*\*\*\*.log c:\Source\connector\XML\*\*\*\*\*\*\*\*.log c:\Source\connector\XML\*\*\*\*\*\*\*\*\*.log]
2020-07-17T17:34:39.020+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\2D\audit.txt
2020-07-17T17:34:39.020+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type2-2D\audit.txt
2020-07-17T17:34:39.021+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2B-Events\audit.txt
2020-07-17T17:34:39.021+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-3C-Events\audit.txt
2020-07-17T17:34:39.022+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-3A\audit.txt
2020-07-17T17:34:39.024+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit__02.txt
2020-07-17T17:34:39.026+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\2A\audit.txt
2020-07-17T17:34:39.026+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-3A-Events\audit.txt
2020-07-17T17:34:39.027+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\3B\audit.txt
2020-07-17T17:34:39.028+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2D\audit.txt
2020-07-17T17:34:39.029+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2D-Events\audit.txt
2020-07-17T17:34:39.030+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2B\audit.txt
2020-07-17T17:34:39.032+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2C-Events\audit.txt
2020-07-17T17:34:39.033+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\2C\audit.txt
2020-07-17T17:34:39.034+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\3D\audit.txt
2020-07-17T17:34:39.036+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2C\audit.txt
2020-07-17T17:34:39.037+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit__05.txt
2020-07-17T17:34:39.038+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit__04.txt
2020-07-17T17:34:39.040+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type2-1D\audit.txt
2020-07-17T17:34:39.041+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit__03.txt
2020-07-17T17:34:39.043+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4ng\ops\audit.txt
2020-07-17T17:34:39.044+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2A\audit.txt
2020-07-17T17:34:39.047+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type2-1\audit.txt
2020-07-17T17:34:39.050+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\2B\audit.txt
2020-07-17T17:34:39.052+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\4A\audit.txt
2020-07-17T17:34:39.054+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit01AMBAL.txt
2020-07-17T17:34:39.056+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type2-2A\audit.txt
2020-07-17T17:34:39.059+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type3-2A-Events\audit.txt
2020-07-17T17:34:39.061+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type1\3A\audit.txt
2020-07-17T17:34:39.065+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4ng\ops\auditAMABL.txt
2020-07-17T17:34:39.067+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type2-2\audit.txt
2020-07-17T17:34:39.068+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit__01.txt
2020-07-17T17:34:39.070+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit02AMBAL.txt
2020-07-17T17:34:39.074+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\audit\type4\ops\audit.txt
2020-07-17T17:34:40.023+0530	INFO	log/harvester.go:216	Harvester started for file: c:\Source\connector\status\type4_v2-1.status
2020-07-17T17:34:40.257+0530	INFO	log/harvester.go:216	Harvester started for file: c:\Source\connector\logs\type4-Schema\connector.pune_type4.log
2020-07-17T17:34:41.846+0530	INFO	log/harvester.go:216	Harvester started for file: c:\source\logs\type3-3C-Events\type3-3C-Events
2020-07-17T17:34:42.793+0530	INFO	log/harvester.go:216	Harvester started for file: c:\Source\connector\CSV\csv_parser_type4NG_VICI20200717_009.log
2020-07-17T17:34:45.030+0530	INFO	log/prospector.go:111	Configured paths: [c:\Source\connector\JSON\*.log c:\Source\connector\JSON\*\*.log c:\Source\connector\JSON\*\*\*.log c:\Source\connector\JSON\*\*\*\*.log c:\Source\connector\JSON\*\*\*\*\*.log c:\Source\connector\JSON\*\*\*\*\*\*.log c:\Source\connector\JSON\*\*\*\*\*\*\*.log c:\Source\connector\JSON\*\*\*\*\*\*\*\*.log c:\Source\connector\JSON\*\*\*\*\*\*\*\*\*.log]
2020-07-17T17:34:48.404+0530	INFO	log/prospector.go:111	Configured paths: [c:\Source\connector\DBConnector\*.log c:\Source\connector\DBConnector\*\*.log c:\Source\connector\DBConnector\*\*\*.log c:\Source\connector\DBConnector\*\*\*\*.log c:\Source\connector\DBConnector\*\*\*\*\*.log c:\Source\connector\DBConnector\*\*\*\*\*\*.log c:\Source\connector\DBConnector\*\*\*\*\*\*\*.log c:\Source\connector\DBConnector\*\*\*\*\*\*\*\*.log c:\Source\connector\DBConnector\*\*\*\*\*\*\*\*\*.log]
2020-07-17T17:34:54.779+0530	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":8562,"time":8656},"total":{"ticks":38312,"time":38796,"value":0},"user":{"ticks":29750,"time":30140}},"info":{"ephemeral_id":"fd675e5f-9417-486e-b698-92a750type36ec5","uptime":{"ms":30726}},"memstats":{"gc_next":11039264,"memory_alloc":5698784,"memory_total":1572943072,"rss":34381824}},"filebeat":{"events":{"active":4186,"added":6438,"done":2252},"harvester":{"open_files":41,"running":38,"started":38}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"active":4096,"batches":2,"total":4096},"read":{"bytes":18},"type":"logstash","write":{"bytes":248701}},"pipeline":{"clients":9,"events":{"active":4123,"filtered":2290,"published":4116,"retry":2048,"total":6413}}},"registrar":{"states":{"cleanup":246,"current":1808,"update":2277},"writes":2276},"system":{"cpu":{"cores":20}}}}}
2020-07-17T17:34:55.823+0530	INFO	log/prospector.go:111	Configured paths: [c:\test\type3*\audit\*.txt c:\test\type4*\audit\*.txt c:\test\type2*\audit\*.txt c:\test\type1\audit\*.txt]
2020-07-17T17:34:55.827+0530	INFO	log/prospector.go:111	Configured paths: [C:\Source\Agent_connector_Heart_Beat\Logs\agent_heartbeat.log]
2020-07-17T17:34:55.828+0530	WARN	[cfgwarn]	cfgfile/reload.go:77	BETA: Dynamic config reload is enabled.
2020-07-17T17:34:55.828+0530	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 10
2020-07-17T17:34:55.828+0530	INFO	cfgfile/reload.go:127	Config reloader started
2020-07-17T17:34:57.592+0530	INFO	log/harvester.go:216	Harvester started for file: c:\Source\connector\XML\type1\xml_parser_type1_3D20200717_047.log
2020-07-17T17:34:59.607+0530	INFO	log/harvester.go:216	Harvester started for file: c:\Source\connector\DBConnector\v2\type21D_ls_217.log
2020-07-17T17:35:06.026+0530	INFO	log/harvester.go:216	Harvester started for file: c:\test\type1\audit\type12b_20200716.txt
2020-07-17T17:35:09.536+0530	INFO	log/harvester.go:216	Harvester started for file: c:\Source\connector\JSON\type3-3A\type33a-line20200717_340.log
2020-07-17T17:35:24.449+0530	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":12609,"time":12609},"total":{"ticks":45905,"time":45905,"value":45905},"user":{"ticks":33296,"time":33296}},"info":{"ephemeral_id":"fd675e5f-9417-486e-b698-92a750type36ec5","uptime":{"ms":60396}},"memstats":{"gc_next":10501232,"memory_alloc":7716744,"memory_total":1736852024,"rss":204800}},"filebeat":{"events":{"active":1,"added":136,"done":135},"harvester":{"open_files":48,"running":42,"started":4}},"libbeat":{"config":{"module":{"running":0}},"output":{"read":{"bytes":36}},"pipeline":{"clients":10,"events":{"active":4125,"filtered":130,"total":132}}},"registrar":{"states":{"cleanup":79,"current":1728,"update":110},"writes":111}}}}
5

You have to add and output to your filebeat configuration.

Althrough your filebeat configuration looks strange, which version of filebeat are you running ?

6

I have Output configured in my Filebeat config. here it is.

output.logstash:
hosts: ["localhost:5045"]

and my filebeat Version: 6.2.3

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.