Hello,
Please help me here.
I have a file needs to be harvested by Filebeats 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File' but its not.
This should work:
filebeat.inputs:
- type: log
paths:
- 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File'
But if you are using a glob then you might be running into this issue: https://github.com/elastic/beats/issues/6737
I appreciate your response in this regard.
I tried exactly as you told before raising this ticket where filebeat was not harvesting the file. I just retried one more time to retry as you suggested and looks like still filebeat is still not harvesting the file. Please note that I cleared off the registry files every time before starting the filebeat so that it harvest files though it is already processed.
But instead of file path 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File’ if I give the path like this its harvesting 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log’.
**Below is my .yml **
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File'
Filebeat Logs:
2018-07-08T10:04:15.887-0500 INFO instance/beat.go:492 Home path: [D:\apps\filebeat] Config path: [D:\apps\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2018-07-08T10:04:15.891-0500 INFO instance/beat.go:499 Beat UUID: 6cba9929-4375-40d7-ac7e-7403146e2595
2018-07-08T10:04:15.892-0500 INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "D:\\apps\\filebeat", "data": "C:\\ProgramData\\filebeat", "home": "D:\\apps\\filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "6cba9929-4375-40d7-ac7e-7403146e2595"}}}
2018-07-08T10:04:15.892-0500 INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "a04cb664d5fbd4b1aab485d1766f3979c138fd38", "libbeat": "6.3.0", "time": "2018-06-11T22:34:03.000Z", "version": "6.3.0"}}}
2018-07-08T10:04:15.892-0500 INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
2018-07-08T10:04:15.899-0500 INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-07-04T17:56:51.81-05:00","hostname":"dwvmlqaci01","ips":["fe80::128:c796:faf8:3236/64","10.32.44.57/23","::1/128","127.0.0.1/8"],"kernel_version":"6.3.9600.19038 (winblue_ltsb_escrow.180608-1416)","mac_addresses":["00:50:56:85:53:13","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19038"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"4a17570a-880e-4fff-b735-9c69f30d9930"}}}
2018-07-08T10:04:15.899-0500 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.0
2018-07-08T10:04:15.900-0500 INFO pipeline/module.go:81 Beat name: dwvmlqaci01
2018-07-08T10:04:15.902-0500 INFO instance/beat.go:315 filebeat start running.
2018-07-08T10:04:15.902-0500 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-07-08T10:04:15.902-0500 INFO registrar/registrar.go:75 No registry file found under: C:\ProgramData\filebeat\registry. Creating a new registry file.
2018-07-08T10:04:15.943-0500 INFO registrar/registrar.go:112 Loading registrar data from C:\ProgramData\filebeat\registry
2018-07-08T10:04:15.943-0500 INFO registrar/registrar.go:123 States Loaded from registrar: 0
2018-07-08T10:04:15.943-0500 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-08T10:04:15.943-0500 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-07-08T10:04:15.944-0500 INFO log/input.go:111 Configured paths: [D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File]
2018-07-08T10:04:15.944-0500 INFO input/input.go:87 Starting input of type: log; ID: 4988709827331553149
2018-07-08T10:04:15.944-0500 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-07-08T10:04:15.944-0500 INFO cfgfile/reload.go:122 Config reloader started
2018-07-08T10:04:45.990-0500 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{"ticks":46,"time":{"ms":46}}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":30049}},"memstats":{"gc_next":4194304,"memory_alloc":1592600,"memory_total":3123176,"rss":16773120}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":8}}}}}
2018-07-08T10:05:15.906-0500 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125},"total":{"ticks":171,"value":171},"user":{"ticks":46}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":60051}},"memstats":{"gc_next”:4194
Thanks
Dinesh
Can you please post a directory listing from the cli too? Like
PS > dir D:\apps\jenkins\jobs\SWX-Regression\builds\180
And I think it would be useful to see the debug logs from Filebeat as it starts up and initializes the inputs and harvesters. If you add logging.level: debug to your config this will enable debug.
Hi Andrew,
Here is the snap of the dir command for the particular directory and also file explorer snap as well
Based on the dir output, the actual file is called log and does not have a file extension so this explains why 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log' works.
Thanks Andrew for the clarification.
I am trying to parse through filbeat as the path configured in .yml as 'D:\apps\jenkins\jobs\SWX-Regression\builds**\log’ but is there any way to parse only the folders that contains only number?
Regards,
Dinesh
@krisdinesh After you succeeded to run the filebeat, do you use logstash and kibana to catch the harvest of filebeat ?
Yes Putra
The glob patterns that are supported are described in detail in the filepath.Match docs from Go. In addition to those patterns we added support for ** to enable recursive matching.
So D:\apps\jenkins\jobs\SWX-Regression\builds\*\log is probably what you want to use. Try that.
Thanks Andrew!!
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.