How to enable filebeat to harvest .File Type?


(Dinesh Krishnaswamy) #1

Hello,

Please help me here.

I have a file needs to be harvested by Filebeats 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File' but its not.


(Andrew Kroh) #2

This should work:

filebeat.inputs:
- type: log
  paths:
    - 'D:\apps\jenkins\jobs\TestJob\builds\180\log.File'

But if you are using a glob then you might be running into this issue: https://github.com/elastic/beats/issues/6737


(Dinesh Krishnaswamy) #3

I appreciate your response in this regard.

I tried exactly as you told before raising this ticket where filebeat was not harvesting the file. I just retried one more time to retry as you suggested and looks like still filebeat is still not harvesting the file. Please note that I cleared off the registry files every time before starting the filebeat so that it harvest files though it is already processed.

But instead of file path 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File’ if I give the path like this its harvesting 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log’.

**Below is my .yml **

#=========================== Filebeat inputs =============================

filebeat.inputs:

# Each - is an input. Most options can be set at the input level, so

# you can use different inputs for various configurations.

# Below are the input specific configurations.

- type: log

# Change to true to enable this input configuration.

enabled: true

# Paths that should be crawled and fetched. Glob based paths.

paths:

- 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File'

Filebeat Logs:

2018-07-08T10:04:15.887-0500	INFO	instance/beat.go:492	Home path: [D:\apps\filebeat] Config path: [D:\apps\filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]

2018-07-08T10:04:15.891-0500	INFO	instance/beat.go:499	Beat UUID: 6cba9929-4375-40d7-ac7e-7403146e2595

2018-07-08T10:04:15.892-0500	INFO	[beat]	instance/beat.go:716	Beat info	{"system_info": {"beat": {"path": {"config": "D:\\apps\\filebeat", "data": "C:\\ProgramData\\filebeat", "home": "D:\\apps\\filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "6cba9929-4375-40d7-ac7e-7403146e2595"}}}

2018-07-08T10:04:15.892-0500	INFO	[beat]	instance/beat.go:725	Build info	{"system_info": {"build": {"commit": "a04cb664d5fbd4b1aab485d1766f3979c138fd38", "libbeat": "6.3.0", "time": "2018-06-11T22:34:03.000Z", "version": "6.3.0"}}}

2018-07-08T10:04:15.892-0500	INFO	[beat]	instance/beat.go:728	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.9.4"}}}

2018-07-08T10:04:15.899-0500	INFO	[beat]	instance/beat.go:732	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-07-04T17:56:51.81-05:00","hostname":"dwvmlqaci01","ips":["fe80::128:c796:faf8:3236/64","10.32.44.57/23","::1/128","127.0.0.1/8"],"kernel_version":"6.3.9600.19038 (winblue_ltsb_escrow.180608-1416)","mac_addresses":["00:50:56:85:53:13","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19038"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"4a17570a-880e-4fff-b735-9c69f30d9930"}}}

2018-07-08T10:04:15.899-0500	INFO	instance/beat.go:225	Setup Beat: filebeat; Version: 6.3.0

2018-07-08T10:04:15.900-0500	INFO	pipeline/module.go:81	Beat name: dwvmlqaci01

2018-07-08T10:04:15.902-0500	INFO	instance/beat.go:315	filebeat start running.

2018-07-08T10:04:15.902-0500	INFO	[monitoring]	log/log.go:97	Starting metrics logging every 30s

2018-07-08T10:04:15.902-0500	INFO	registrar/registrar.go:75	No registry file found under: C:\ProgramData\filebeat\registry. Creating a new registry file.

2018-07-08T10:04:15.943-0500	INFO	registrar/registrar.go:112	Loading registrar data from C:\ProgramData\filebeat\registry

2018-07-08T10:04:15.943-0500	INFO	registrar/registrar.go:123	States Loaded from registrar: 0

2018-07-08T10:04:15.943-0500	WARN	beater/filebeat.go:354	Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.

2018-07-08T10:04:15.943-0500	INFO	crawler/crawler.go:48	Loading Inputs: 1

2018-07-08T10:04:15.944-0500	INFO	log/input.go:111	Configured paths: [D:\apps\jenkins\jobs\SWX-Regression\builds\180\log.File]

2018-07-08T10:04:15.944-0500	INFO	input/input.go:87	Starting input of type: log; ID: 4988709827331553149

2018-07-08T10:04:15.944-0500	INFO	crawler/crawler.go:82	Loading and starting Inputs completed. Enabled inputs: 1

2018-07-08T10:04:15.944-0500	INFO	cfgfile/reload.go:122	Config reloader started

2018-07-08T10:04:45.990-0500	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125,"time":{"ms":125}},"total":{"ticks":171,"time":{"ms":171},"value":171},"user":{"ticks":46,"time":{"ms":46}}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":30049}},"memstats":{"gc_next":4194304,"memory_alloc":1592600,"memory_total":3123176,"rss":16773120}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":3},"output":{"type":"logstash"},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":1},"system":{"cpu":{"cores":8}}}}}

2018-07-08T10:05:15.906-0500	INFO	[monitoring]	log/log.go:124	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":125},"total":{"ticks":171,"value":171},"user":{"ticks":46}},"info":{"ephemeral_id":"ba8f2ab1-ac14-4d1d-8ca6-b0bc9c314c72","uptime":{"ms":60051}},"memstats":{"gc_next”:4194

Thanks

Dinesh


(Andrew Kroh) #4

Can you please post a directory listing from the cli too? Like

PS > dir D:\apps\jenkins\jobs\SWX-Regression\builds\180

And I think it would be useful to see the debug logs from Filebeat as it starts up and initializes the inputs and harvesters. If you add logging.level: debug to your config this will enable debug.


(Dinesh Krishnaswamy) #5

Hi Andrew,

Here is the snap of the dir command for the particular directory and also file explorer snap as well


(Andrew Kroh) #6

Based on the dir output, the actual file is called log and does not have a file extension so this explains why 'D:\apps\jenkins\jobs\SWX-Regression\builds\180\log' works.


(Dinesh Krishnaswamy) #7

Thanks Andrew for the clarification.

I am trying to parse through filbeat as the path configured in .yml as 'D:\apps\jenkins\jobs\SWX-Regression\builds**\log’ but is there any way to parse only the folders that contains only number?

Regards,

Dinesh


(Wahyu Putra) #8

@krisdinesh After you succeeded to run the filebeat, do you use logstash and kibana to catch the harvest of filebeat ?


(Dinesh Krishnaswamy) #9

Yes Putra


(Andrew Kroh) #10

The glob patterns that are supported are described in detail in the filepath.Match docs from Go. In addition to those patterns we added support for ** to enable recursive matching.

So D:\apps\jenkins\jobs\SWX-Regression\builds\*\log is probably what you want to use. Try that.


(Dinesh Krishnaswamy) #11

Thanks Andrew!!


(system) #12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.