Hello,
Please help me here.
I have a file needs to be harvested by Filebeats 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File' but its not.
This should work:
filebeat.inputs:
- type: log
paths:
- 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File'
But if you are using a glob then you might be running into this issue: https://github.com/elastic/beats/issues/6737
I appreciate your response in this regard.
I tried exactly as you told before raising this ticket where filebeat was not harvesting the file. I just retried one more time to retry as you suggested and looks like still filebeat is still not harvesting the file. Please note that I cleared off the registry files every time before starting the filebeat so that it harvest files though it is already processed.
But instead of file path 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File’ if I give the path like this its harvesting 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log’.
**Below is my .yml **
#=========================== Filebeat inputs =============================
filebeat.inputs:
# Each - is an input. Most options can be set at the input level, so
# you can use different inputs for various configurations.
# Below are the input specific configurations.
- type: log
# Change to true to enable this input configuration.
enabled: true
# Paths that should be crawled and fetched. Glob based paths.
paths:
- 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File'
Filebeat Logs:
2018-07-08T10:04:15.887-0500 INFO instance/beat.go:492 Home path: [D:\apps\filebeat] Config path: [D:\apps\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2018-07-08T10:04:15.891-0500 INFO instance/beat.go:499 Beat UUID: 6cba9929-4375-40d7-ac7e-7403146e2595
2018-07-08T10:04:15.892-0500 INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "D:\\apps\\filebeat", "data": "C:\\ProgramData\\filebeat", "home": "D:\\apps\\filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "6cba9929-4375-40d7-ac7e-7403146e2595"}}}
2018-07-08T10:04:15.892-0500 INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "a04cb664d5fbd4b1aab485d1766f3979c138fd38", "libbeat": "6.3.0", "time": "2018-06-11T22:34:03.000Z", "version": "6.3.0"}}}
2018-07-08T10:04:15.892-0500 INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}
2018-07-08T10:04:15.899-0500 INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-07-04T17:56:51.81-05:00","hostname":"dwvmlqaci01","ips":["fe80::128:c796:faf8:3236/64","10.32.44.57/23","::1/128","127.0.0.1/8"],"kernel_version":"6.3.9600.19038 (winblue_ltsb_escrow.180608-1416)","mac_addresses":["00:50:56:85:53:13","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19038"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"4a17570a-880e-4fff-b735-9c69f30d9930"}}}
2018-07-08T10:04:15.899-0500 INFO instance/beat.go:225 Setup Beat: filebeat; Version: 6.3.0
2018-07-08T10:04:15.900-0500 INFO pipeline/module.go:81 Beat name: dwvmlqaci01
2018-07-08T10:04:15.902-0500 INFO instance/beat.go:315 filebeat start running.
2018-07-08T10:04:15.902-0500 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-07-08T10:04:15.902-0500 INFO registrar/registrar.go:75 No registry file found under: C:\ProgramData\filebeat\registry. Creating a new registry file.
2018-07-08T10:04:15.943-0500 INFO registrar/registrar.go:112 Loading registrar data from C:\ProgramData\filebeat\registry
2018-07-08T10:04:15.943-0500 INFO registrar/registrar.go:123 States Loaded from registrar: 0
2018-07-08T10:04:15.943-0500 WARN beater/filebeat.go:354 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2018-07-08T10:04:15.943-0500 INFO crawler/crawler.go:48 Loading Inputs: 1
2018-07-08T10:04:15.944-0500 INFO log/input.go:111 Configured paths: [D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File]
2018-07-08T10:04:15.944-0500 INFO input/input.go:87 Starting input of type: log; ID: 4988709827331553149
2018-07-08T10:04:15.944-0500 INFO crawler/crawler.go:82 Loading and starting Inputs completed. Enabled inputs: 1
2018-07-08T10:04:15.944-0500 INFO cfgfile/reload.go:122 Config reloader started
2018-07-08T10:04:45.990-0500 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{"ticks":46,"time":{"ms":46}}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":30049}},"memstats":{"gc_next":4194304,"memory_alloc":1592600,"memory_total":3123176,"rss":16773120}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":8}}}}}
2018-07-08T10:05:15.906-0500 INFO [monitoring] log/log.go:124 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125},"total":{"ticks":171,"value":171},"user":{"ticks":46}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":60051}},"memstats":{"gc_next”:4194
Thanks
Dinesh
Can you please post a directory listing from the cli too? Like
PS > dir D:\apps\jenkins\jobs\SWX-Regression\builds\180
And I think it would be useful to see the debug logs from Filebeat as it starts up and initializes the inputs and harvesters. If you add logging.level: debug to your config this will enable debug.
Based on the dir output, the actual file is called log and does not have a file extension so this explains why 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log' works.
Thanks Andrew for the clarification.
I am trying to parse through filbeat as the path configured in .yml as 'D:\apps\jenkins\jobs\SWX-Regression\builds**\log’ but is there any way to parse only the folders that contains only number?
Regards,
Dinesh
@krisdinesh After you succeeded to run the filebeat, do you use logstash and kibana to catch the harvest of filebeat ?
Yes Putra
The glob patterns that are supported are described in detail in the filepath.Match docs from Go. In addition to those patterns we added support for ** to enable recursive matching.
So D:\apps\jenkins\jobs\SWX-Regression\builds\*\log is probably what you want to use. Try that.
Thanks Andrew!!