filebeat.yml:
path.config: /Users/sp/servers/filebeat-7.14.0-darwin-x86_64/
filebeat.config:
inputs:
enabled: false
path: /Users/sp/servers/filebeat-7.14.0-darwin-x86_64/filebeat.yml
filebeat.inputs:
- type: filestream
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /Users/sp/beewise_data*.txt2
filebeat.config.modules:
# Glob pattern for configuration loading
path: ${path.config}/modules.d/*.yml
# Set to true to enable config reloading
reload.enabled: false
# Period on which files under path should be checked for changes
#reload.period: 10s
# ------------------------------ Logstash Output -------------------------------
output.logstash:
# The Logstash hosts
hosts: ["localhost:5044"]
I have mentioned a very specific path, but still Filebeat harvests log files:
2021-08-17T11:14:57.964+0530 INFO instance/beat.go:665 Home path: [/Users/sp/servers/filebeat-7.14.0-darwin-x86_64] Config path: [/Users/sp/servers/filebeat-7.14.0-darwin-x86_64/] Data path: [/Users/sp/servers/filebeat-7.14.0-darwin-x86_64/data] Logs path: [/Users/sp/servers/filebeat-7.14.0-darwin-x86_64/logs]
2021-08-17T11:14:57.966+0530 INFO instance/beat.go:673 Beat ID: 375332d9-ad7e-b561-41ebf10993d6
2021-08-17T11:14:57.967+0530 INFO [beat] instance/beat.go:1014 Beat info {"system_info": {"beat": {"path": {"config": "/Users/sp/servers/filebeat-7.14.0-darwin-x86_64", "data": "/Users/sp/servers/filebeat-7.14.0-darwin-x86_64/data", "home": "/Users/sp/servers/filebeat-7.14.0-darwin-x86_64", "logs": "/Users/sp/servers/filebeat-7.14.0-darwin-x86_64/logs"}, "type": "filebeat", "uuid": "375332d9-ad7e-b561-41ebf10993d6"}}}
2021-08-17T11:14:57.967+0530 INFO [beat] instance/beat.go:1023 Build info {"system_info": {"build": {"commit": "e127fc31fc6c00fdf8649808f9421d8f8c28b5db", "libbeat": "7.14.0", "time": "2021-07-29T20:57:00.000Z", "version": "7.14.0"}}}
2021-08-17T11:14:57.967+0530 INFO [beat] instance/beat.go:1026 Go runtime info {"system_info": {"go": {"os":"darwin","arch":"amd64","max_procs":8,"version":"go1.16.6"}}}
2021-08-17T11:14:57.967+0530 INFO [beat] instance/beat.go:1030 Host info {...}
2021-08-17T11:14:57.968+0530 INFO [beat] instance/beat.go:1059 Process info {"system_info": {"process": {"cwd": "/Users/sp/servers/filebeat-7.14.0-darwin-x86_64", "exe": "./filebeat", "name": "filebeat", "pid": 46641, "ppid": 80974, "start_time": "2021-08-17T11:14:57.471+0530"}}}
2021-08-17T11:14:57.968+0530 INFO instance/beat.go:309 Setup Beat: filebeat; Version: 7.14.0
2021-08-17T11:14:57.970+0530 INFO [publisher] pipeline/module.go:113 Beat name: ip-192-168-1-3.ap-south-1.compute.internal
2021-08-17T11:14:57.972+0530 WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-08-17T11:14:57.973+0530 INFO instance/beat.go:473 filebeat start running.
2021-08-17T11:14:57.973+0530 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-08-17T11:14:57.973+0530 INFO memlog/store.go:119 Loading data file of '/Users/sp/servers/filebeat-7.14.0-darwin-x86_64/data/registry/filebeat' succeeded. Active transaction id=0
2021-08-17T11:14:57.975+0530 INFO memlog/store.go:124 Finished loading transaction log file for '/Users/sp/servers/filebeat-7.14.0-darwin-x86_64/data/registry/filebeat'. Active transaction id=2
2021-08-17T11:14:57.975+0530 WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2021-08-17T11:14:57.977+0530 INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 0
2021-08-17T11:14:57.978+0530 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2021-08-17T11:14:57.979+0530 INFO [crawler] beater/crawler.go:141 Starting input (ID: 105059798570908)
2021-08-17T11:14:57.979+0530 INFO [input.filestream] compat/compat.go:111 Input filestream starting {"id": "91CCB6C2C59C"}
2021-08-17T11:14:57.979+0530 INFO [file_watcher] filestream/fswatch.go:137 Start next scan
2021-08-17T11:14:57.986+0530 INFO [input] log/input.go:164 Configured paths: [/var/log/secure.log*] {"input_id": "b71bf4e8-6f99-844a-18789f49e42e"}
2021-08-17T11:14:57.986+0530 INFO [input] log/input.go:164 Configured paths: [/var/log/system.log*] {"input_id": "c05812b8-f776-834d-73c7da9bb4ef"}
2021-08-17T11:14:57.986+0530 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2021-08-17T11:14:57.987+0530 INFO cfgfile/reload.go:164 Config reloader started
2021-08-17T11:14:57.993+0530 INFO [input] log/input.go:164 Configured paths: [/var/log/secure.log*] {"input_id": "51291c2f-4988-8f18-fb78273b5b6e"}
2021-08-17T11:14:57.994+0530 INFO [input] log/input.go:164 Configured paths: [/var/log/system.log*] {"input_id": "ee9bbd79-279a-971d-398bcd02068a"}
2021-08-17T11:14:57.994+0530 INFO cfgfile/reload.go:224 Loading of config files completed.
2021-08-17T11:14:57.998+0530 INFO [input.harvester] log/harvester.go:309 Harvester started for file. {"input_id": "ee9bbd79-279a-971d-398bcd02068a", "source": "/var/log/system.log", "state_id": "native::10820-16777220", "finished": false, "os_id": "10820-167720", "harvester_id": "b01ae8f9-1469-ae2d-b03e053118f4"}