Filebeat connect to Elasticsearch failed

Elastic Stack Beats
1

Hi,

I have a Win10 for client and ELK ubuntu for server.
I have installed "Winlogbeat" and "Filebeat" on the client simultaneously. And, I have make sure the connect between "Winlogbeat" and "Elasticsearch" is correct that the event logs can be shown on Kibana.

Then, I let the "filebeat.yml" setting same as "winlogbeat.yml". Just as below:
"ELK host ip" is my ELK host ip address.

#============================== Kibana =====================================
setup.kibana:
  host: "ELK host ip:5601"
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["ELK host ip:9200"]

Then, here is my "filebeat.yml" input.

#========== Filebeat inputs ==========
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\testcase\*.log

I had tried the command ".\filebeat test config .\filebeat.yml" and get the result of "Config OK".
Then, I make some testing logs in the path and "Start-Service Filebeat".

But, Kibana can't see any message about "Filebeat". How do I solve it?

2

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them :slight_smile:

What do the Filebeat logs show once it's started?

3

I'm sorry about that.

Here is my Filebeat log.

2020-05-26T15:08:15.846+0800	INFO	instance/beat.go:621	Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2020-05-26T15:08:15.884+0800	INFO	instance/beat.go:629	Beat ID: b0535027-60e5-4d11-8178-2a678de322d0
2020-05-26T15:08:15.896+0800	INFO	[beat]	instance/beat.go:957	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Filebeat", "data": "C:\\Program Files\\Filebeat\\data", "home": "C:\\Program Files\\Filebeat", "logs": "C:\\Program Files\\Filebeat\\logs"}, "type": "filebeat", "uuid": "b0535027-60e5-4d11-8178-2a678de322d0"}}}
2020-05-26T15:08:15.896+0800	INFO	[beat]	instance/beat.go:966	Build info	{"system_info": {"build": {"commit": "5e69e25b920e3d93bec76a09a31da3ab35a55607", "libbeat": "7.7.0", "time": "2020-05-12T00:53:14.000Z", "version": "7.7.0"}}}
2020-05-26T15:08:15.896+0800	INFO	[beat]	instance/beat.go:969	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.13.9"}}}
2020-05-26T15:08:15.902+0800	INFO	[beat]	instance/beat.go:973	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-25T07:21:40.25+08:00","name":"Stan-VM","ip":["fe80::d90a:ca:694d:a307/64","[Client host ip]/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.836 (WinBuild.160101.0800)","mac":["00:0c:29:5b:ab:7c"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.836"},"timezone":"CST","timezone_offset_sec":28800,"id":"31552a7a-6e38-4d05-9b26-009a907b3320"}}}
2020-05-26T15:08:15.920+0800	INFO	[beat]	instance/beat.go:1002	Process info	{"system_info": {"process": {"cwd": "C:\\Program Files\\Filebeat", "exe": "C:\\Program Files\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 8912, "ppid": 7568, "start_time": "2020-05-26T15:08:15.218+0800"}}}
2020-05-26T15:08:15.920+0800	INFO	instance/beat.go:297	Setup Beat: filebeat; Version: 7.7.0
2020-05-26T15:08:15.920+0800	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'filebeat-7.7.0' as ILM is enabled.
2020-05-26T15:08:15.920+0800	INFO	eslegclient/connection.go:84	elasticsearch url: http://[ELK host ip]:9200
2020-05-26T15:08:15.920+0800	INFO	[publisher]	pipeline/module.go:110	Beat name: Stan-VM
4

I had used the command on PowerShell :

.\filebeat.exe -c .\filebeat.yml -e -d "*"

It seems that Filebeat is work correctly on my local client. And, here is my Filebeat logs.
I put a empty log named ,"ttttttt.log".

2020-05-28T13:53:12.098+0800	INFO	instance/beat.go:622	Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2020-05-28T13:53:12.128+0800	INFO	instance/beat.go:630	Beat ID: ae2d8b74-998f-4903-b9aa-12c3d6e32045
2020-05-28T13:53:12.140+0800	INFO	[beat]	instance/beat.go:958	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Filebeat", "data": "C:\\ProgramData\\filebeat", "home": "C:\\Program Files\\Filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "ae2d8b74-998f-4903-b9aa-12c3d6e32045"}}}
2020-05-28T13:53:12.140+0800	INFO	[beat]	instance/beat.go:967	Build info	{"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:23:36.000Z", "version": "7.6.2"}}}
2020-05-28T13:53:12.140+0800	INFO	[beat]	instance/beat.go:970	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.13.8"}}}
2020-05-28T13:53:12.146+0800	INFO	[beat]	instance/beat.go:974	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-28T12:02:42.78+08:00","name":"Stan-VM","ip":["fe80::d90a:ca:694d:a307/64","10.3.87.201/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.836 (WinBuild.160101.0800)","mac":["00:0c:29:5b:ab:7c"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.836"},"timezone":"CST","timezone_offset_sec":28800,"id":"31552a7a-6e38-4d05-9b26-009a907b3320"}}}
2020-05-28T13:53:12.150+0800	INFO	[beat]	instance/beat.go:1003	Process info	{"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "C:\\Program Files\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 4232, "ppid": 680, "start_time": "2020-05-28T13:53:11.508+0800"}}}
2020-05-28T13:53:12.150+0800	INFO	instance/beat.go:298	Setup Beat: filebeat; Version: 7.6.2
2020-05-28T13:53:12.150+0800	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'filebeat-7.6.2' as ILM is enabled.
2020-05-28T13:53:12.151+0800	INFO	elasticsearch/client.go:174	Elasticsearch url: http://[ELK host ip]:9200
2020-05-28T13:53:12.151+0800	INFO	[publisher]	pipeline/module.go:110	Beat name: Stan-VM
2020-05-28T13:53:12.152+0800	INFO	instance/beat.go:439	filebeat start running.
2020-05-28T13:53:12.152+0800	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-05-28T13:53:12.158+0800	INFO	registrar/registrar.go:145	Loading registrar data from C:\ProgramData\filebeat\registry\filebeat\data.json
2020-05-28T13:53:12.158+0800	INFO	registrar/registrar.go:152	States Loaded from registrar: 2
2020-05-28T13:53:12.158+0800	INFO	crawler/crawler.go:72	Loading Inputs: 1
2020-05-28T13:53:12.158+0800	INFO	log/input.go:152	Configured paths: [C:\Users\Stan_VM_Win10_1\Desktop\testcase\*]
2020-05-28T13:53:12.158+0800	INFO	input/input.go:114	Starting input of type: log; ID: 10865679355279554090 
2020-05-28T13:53:12.158+0800	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2020-05-28T13:53:12.160+0800	INFO	cfgfile/reload.go:175	Config reloader started
2020-05-28T13:53:12.160+0800	INFO	cfgfile/reload.go:235	Loading of config files completed.
2020-05-28T13:53:15.151+0800	INFO	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-05-28T13:53:42.154+0800	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78,"time":{"ms":78}},"total":{"ticks":109,"time":{"ms":109},"value":109},"user":{"ticks":31,"time":{"ms":31}}},"handles":{"open":249},"info":{"ephemeral_id":"aad0228c-23ec-408c-a7b4-fcd0e0f9deb0","uptime":{"ms":30103}},"memstats":{"gc_next":11883840,"memory_alloc":8270072,"memory_total":13680456,"rss":36716544},"runtime":{"goroutines":25}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":2,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}}
2020-05-28T13:54:12.154+0800	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78},"total":{"ticks":109,"value":109},"user":{"ticks":31}},"handles":{"open":247},"info":{"ephemeral_id":"aad0228c-23ec-408c-a7b4-fcd0e0f9deb0","uptime":{"ms":60104}},"memstats":{"gc_next":11883840,"memory_alloc":8353680,"memory_total":13764064,"rss":36864},"runtime":{"goroutines":25}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2020-05-28T13:54:22.166+0800	INFO	log/harvester.go:297	Harvester started for file: C:\Users\Stan_VM_Win10_1\Desktop\testcase\ttttttt.log
2020-05-28T13:54:42.153+0800	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":93,"time":{"ms":15}},"total":{"ticks":124,"time":{"ms":15},"value":124},"user":{"ticks":31}},"handles":{"open":248},"info":{"ephemeral_id":"aad0228c-23ec-408c-a7b4-fcd0e0f9deb0","uptime":{"ms":90103}},"memstats":{"gc_next":11883840,"memory_alloc":8466784,"memory_total":13877168,"rss":90112},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"files":{"72369424-3364-40d4-a720-c3c23f0b0cad":{"last_event_published_time":"","last_event_timestamp":"","name":"C:\\Users\\Stan_VM_Win10_1\\Desktop\\testcase\\ttttttt.log","start_time":"2020-05-28T13:54:22.165Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":3,"update":1},"writes":{"success":1,"total":1}}}}}

How do I sent this message to the Elasticsearch?

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.