Filebeat adding incorrect path to multiple prospectors?


(magdiel ) #1

Hi I just started using filebeat I am using Filebeat - 6.2.4. I am running this on a redhat 5 linux box and I am trying to send around 15 log files into elasticsearch. Not sure where I am going wrong in specifying the paths for these in the filebeat.yml file. In the logs it shows that filebeat is adding its own path to the rest of the prospectors I created other than the first. Help would be greatly appreciated.

Here is my filebeat.yml file:

#=========================== Filebeat prospectors =============================

filebeat.prospectors:

# Each - is a prospector. Most options can be set at the prospector level, so
# you can use different prospectors for various configurations.
# Below are the prospector specific configurations.

- type: log

  # Change to true to enable this prospector configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /opt/bs/apps/readerstreet/current/logs/Readerstreet.*.report

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures1/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures2/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures3/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures4/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures5/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures6/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures7/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures8/bin/*.log

- type: log

  paths:
    -/opt/bs/apps/BrainProcedures9/bin/*.log

and In my log file that shows the results of filebeat here is what it shows for the multiple prospectors:

2018-06-01T09:17:56.539-0500	INFO	instance/beat.go:468	Home path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64] Config path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64] Data path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/data] Logs path: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/logs]
2018-06-01T09:17:56.543-0500	INFO	instance/beat.go:475	Beat UUID: aabea580-3d3b-4a32-8750-abac7795fd7f
2018-06-01T09:17:56.543-0500	INFO	instance/beat.go:213	Setup Beat: filebeat; Version: 6.2.4
2018-06-01T09:17:56.543-0500	INFO	elasticsearch/client.go:145	Elasticsearch url: http://xldogsa18:9400
2018-06-01T09:17:56.544-0500	INFO	pipeline/module.go:76	Beat name: xldogs10
2018-06-01T09:17:56.619-0500	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s
2018-06-01T09:17:56.620-0500	INFO	instance/beat.go:301	filebeat start running.
2018-06-01T09:17:56.622-0500	INFO	registrar/registrar.go:110	Loading registrar data from /opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/data/registry
2018-06-01T09:17:56.623-0500	INFO	registrar/registrar.go:121	States Loaded from registrar: 5
2018-06-01T09:17:56.623-0500	INFO	crawler/crawler.go:48	Loading Prospectors: 10
2018-06-01T09:17:56.644-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/apps/readerstreet/current/logs/Readerstreet.*.report]
2018-06-01T09:17:56.651-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures1/bin/*.log]
2018-06-01T09:17:56.667-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures2/bin/*.log]
2018-06-01T09:17:56.668-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures3/bin/*.log]
2018-06-01T09:17:56.668-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures4/bin/*.log]
2018-06-01T09:17:56.679-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures5/bin/*.log]
2018-06-01T09:17:56.680-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures6/bin/*.log]
2018-06-01T09:17:56.682-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures7/bin/*.log]
2018-06-01T09:17:56.682-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures8/bin/*.log]
2018-06-01T09:17:56.683-0500	INFO	log/prospector.go:111	Configured paths: [/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures9/bin/*.log]
2018-06-01T09:17:56.714-0500	INFO	log/harvester.go:216	Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.DIG12.report
2018-06-01T09:17:56.720-0500	INFO	crawler/crawler.go:82	Loading and starting Prospectors completed. Enabled prospectors: 10
2018-06-01T09:17:56.720-0500	INFO	log/harvester.go:216	Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.xl12.report
2018-06-01T09:17:56.720-0500	INFO	cfgfile/reload.go:127	Config reloader started
2018-06-01T09:17:56.721-0500	INFO	cfgfile/reload.go:219	Loading of config files completed.
2018-06-01T09:17:56.721-0500	INFO	log/harvester.go:216	Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.xl113.report
2018-06-01T09:17:56.723-0500	INFO	log/harvester.go:216	Harvester started for file: /opt/bs/apps/readerstreet/current/logs/Readerstreet.xl12.report
2018-06-01T09:17:57.744-0500	INFO	elasticsearch/client.go:690	Connected to Elasticsearch version 2.3.5
2018-06-01T09:17:58.110-0500	INFO	template/load.go:73	Template already exists and will not be overwritten.
2018-06-01T09:18:26.638-0500	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	

I notice that for the first prospector it does have the right configured path but for the rest it adds the path that I have filebeat in infront of them. (ie "/opt/bs/filebeat/filebeat-6.2.4-linux-x86_64/-/opt/bs/apps/BrainProcedures1/bin/*.log") How can I change that? Thank you


(ruflin) #2

Can you add a space in all your configs after the - and between the path?


(magdiel ) #3

That was the issue. Thank you Ruflin!


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.