How to enable filebeat to harvest .File Type?


Please help me here.

I have a file needs to be harvested by Filebeats 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File' but its not.

This should work:

- type: log
    - 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File'

But if you are using a glob then you might be running into this issue:

I appreciate your response in this regard.

I tried exactly as you told before raising this ticket where filebeat was not harvesting the file. I just retried one more time to retry as you suggested and looks like still filebeat is still not harvesting the file. Please note that I cleared off the registry files every time before starting the filebeat so that it harvest files though it is already processed.

But instead of file path 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File’ if I give the path like this its harvesting 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log’.

**Below is my .yml **

#=========================== Filebeat inputs =============================


# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

- type: log

# Change to true to enable this input configuration.

enabled: true

# Paths that should be crawled and fetched. Glob based paths.


- 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File'

Filebeat Logs:

2018-07-08T10:04:15.887-0500	INFO	instance/beat.go:492	Home path: [D:\apps\filebeat] Config path: [D:\apps\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]

2018-07-08T10:04:15.891-0500	INFO	instance/beat.go:499	Beat UUID: 6cba9929-4375-40d7-ac7e-7403146e2595

2018-07-08T10:04:15.892-0500	INFO	[beat]	instance/beat.go:716	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\apps\\filebeat", "data": "C:\\ProgramData\\filebeat", "home": "D:\\apps\\filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "6cba9929-4375-40d7-ac7e-7403146e2595"}}}

2018-07-08T10:04:15.892-0500	INFO	[beat]	instance/beat.go:725	Build info	{"system_info": {"build": {"commit": "a04cb664d5fbd4b1aab485d1766f3979c138fd38", "libbeat": "6.3.0", "time": "2018-06-11T22:34:03.000Z", "version": "6.3.0"}}}

2018-07-08T10:04:15.892-0500	INFO	[beat]	instance/beat.go:728	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}

2018-07-08T10:04:15.899-0500	INFO	[beat]	instance/beat.go:732	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-07-04T17:56:51.81-05:00","hostname":"dwvmlqaci01","ips":["fe80::128:c796:faf8:3236/64","","::1/128",""],"kernel_version":"6.3.9600.19038 (winblue_ltsb_escrow.180608-1416)","mac_addresses":["00:50:56:85:53:13","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19038"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"4a17570a-880e-4fff-b735-9c69f30d9930"}}}

2018-07-08T10:04:15.899-0500	INFO	instance/beat.go:225	Setup Beat: filebeat; Version: 6.3.0

2018-07-08T10:04:15.900-0500	INFO	pipeline/module.go:81	Beat name: dwvmlqaci01

2018-07-08T10:04:15.902-0500	INFO	instance/beat.go:315	filebeat start running.

2018-07-08T10:04:15.902-0500	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s

2018-07-08T10:04:15.902-0500	INFO	registrar/registrar.go:75	No registry file found under: C:\ProgramData\filebeat\registry. Creating a new registry file.

2018-07-08T10:04:15.943-0500	INFO	registrar/registrar.go:112	Loading registrar data from C:\ProgramData\filebeat\registry

2018-07-08T10:04:15.943-0500	INFO	registrar/registrar.go:123	States Loaded from registrar: 0

2018-07-08T10:04:15.943-0500	WARN	beater/filebeat.go:354	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

2018-07-08T10:04:15.943-0500	INFO	crawler/crawler.go:48	Loading Inputs: 1

2018-07-08T10:04:15.944-0500	INFO	log/input.go:111	Configured paths: [D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File]

2018-07-08T10:04:15.944-0500	INFO	input/input.go:87	Starting input of type: log; ID: 4988709827331553149

2018-07-08T10:04:15.944-0500	INFO	crawler/crawler.go:82	Loading and starting Inputs completed. Enabled inputs: 1

2018-07-08T10:04:15.944-0500	INFO	cfgfile/reload.go:122	Config reloader started

2018-07-08T10:04:45.990-0500	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{"ticks":46,"time":{"ms":46}}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":30049}},"memstats":{"gc_next":4194304,"memory_alloc":1592600,"memory_total":3123176,"rss":16773120}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":8}}}}}

2018-07-08T10:05:15.906-0500	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125},"total":{"ticks":171,"value":171},"user":{"ticks":46}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":60051}},"memstats":{"gc_next”:4194



Can you please post a directory listing from the cli too? Like

PS > dir D:\apps\jenkins\jobs\SWX-Regression\builds\180

And I think it would be useful to see the debug logs from Filebeat as it starts up and initializes the inputs and harvesters. If you add logging.level: debug to your config this will enable debug.

Hi Andrew,

Here is the snap of the dir command for the particular directory and also file explorer snap as well

Based on the dir output, the actual file is called log and does not have a file extension so this explains why 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log' works.

Thanks Andrew for the clarification.

I am trying to parse through filbeat as the path configured in .yml as 'D:\apps\jenkins\jobs\SWX-Regression\builds**\log’ but is there any way to parse only the folders that contains only number?



@krisdinesh After you succeeded to run the filebeat, do you use logstash and kibana to catch the harvest of filebeat ?

Yes Putra

The glob patterns that are supported are described in detail in the filepath.Match docs from Go. In addition to those patterns we added support for ** to enable recursive matching.

So D:\apps\jenkins\jobs\SWX-Regression\builds\*\log is probably what you want to use. Try that.

Thanks Andrew!!

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.