Hi I just started using filebeat I am using Filebeat - 6.2.4. I am running this on a redhat 5 linux box and I am trying to send around 15 log files into elasticsearch. Not sure where I am going wrong in specifying the paths for these in the filebeat.yml file. In the logs it shows that filebeat is adding its own path to the rest of the prospectors I created other than the first. Help would be greatly appreciated.
Here is my filebeat.yml file:
#=========================== Filebeat prospectors =============================
filebeat.prospectors:
# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.
- type: log
# Change to true to enable this prospector configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- /opt/bs/apps/readerstreet/current/logs/Readerstreet.*.report
- type: log
paths:
-/opt/bs/apps/BrainProcedures1/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures2/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures3/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures4/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures5/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures6/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures7/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures8/bin/*.log
- type: log
paths:
-/opt/bs/apps/BrainProcedures9/bin/*.log
and In my log file that shows the results of filebeat here is what it shows for the multiple prospectors:
2018-06-01T09:17:56.539-0500 INFO instance/beat.go:468 Home path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/logs]
2018-06-01T09:17:56.543-0500 INFO instance/beat.go:475 Beat UUID: aabea580-3d3b-4a32-8750-abac7795fd7f
2018-06-01T09:17:56.543-0500 INFO instance/beat.go:213 Setup Beat: filebeat; Version: 6.2.4
2018-06-01T09:17:56.543-0500 INFO elasticsearch/client.go:145 Elasticsearch url: http://xldogsa18:9400
2018-06-01T09:17:56.544-0500 INFO pipeline/module.go:76 Beat name: xldogs10
2018-06-01T09:17:56.619-0500 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-06-01T09:17:56.620-0500 INFO instance/beat.go:301 filebeat start running.
2018-06-01T09:17:56.622-0500 INFO registrar/registrar.go:110 Loading registrar data from /opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/data/registry
2018-06-01T09:17:56.623-0500 INFO registrar/registrar.go:121 States Loaded from registrar: 5
2018-06-01T09:17:56.623-0500 INFO crawler/crawler.go:48 Loading Prospectors: 10
2018-06-01T09:17:56.644-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/apps/readerstreet/current/logs/Readerstreet.*.report]
2018-06-01T09:17:56.651-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures1/bin/*.log]
2018-06-01T09:17:56.667-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures2/bin/*.log]
2018-06-01T09:17:56.668-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures3/bin/*.log]
2018-06-01T09:17:56.668-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures4/bin/*.log]
2018-06-01T09:17:56.679-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures5/bin/*.log]
2018-06-01T09:17:56.680-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures6/bin/*.log]
2018-06-01T09:17:56.682-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures7/bin/*.log]
2018-06-01T09:17:56.682-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures8/bin/*.log]
2018-06-01T09:17:56.683-0500 INFO log/prospector.go:111 Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures9/bin/*.log]
2018-06-01T09:17:56.714-0500 INFO log/harvester.go:216 Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.DIG12.report
2018-06-01T09:17:56.720-0500 INFO crawler/crawler.go:82 Loading and starting Prospectors completed. Enabled prospectors: 10
2018-06-01T09:17:56.720-0500 INFO log/harvester.go:216 Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.xl12.report
2018-06-01T09:17:56.720-0500 INFO cfgfile/reload.go:127 Config reloader started
2018-06-01T09:17:56.721-0500 INFO cfgfile/reload.go:219 Loading of config files completed.
2018-06-01T09:17:56.721-0500 INFO log/harvester.go:216 Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.xl113.report
2018-06-01T09:17:56.723-0500 INFO log/harvester.go:216 Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.xl12.report
2018-06-01T09:17:57.744-0500 INFO elasticsearch/client.go:690 Connected to Elasticsearch version 2.3.5
2018-06-01T09:17:58.110-0500 INFO template/load.go:73 Template already exists and will not be overwritten.
2018-06-01T09:18:26.638-0500 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s
I notice that for the first prospector it does have the right configured path but for the rest it adds the path that I have filebeat in infront of them. (ie "/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures1/bin/*.log") How can I change that? Thank you