Filebeat connect to Elasticsearch failed

[stanwang]

Hi,

I have a Win10 for client and ELK ubuntu for server.
I have installed "Winlogbeat" and "Filebeat" on the client simultaneously. And, I have make sure the connect between "Winlogbeat" and "Elasticsearch" is correct that the event logs can be shown on Kibana.

Then, I let the "filebeat.yml" setting same as "winlogbeat.yml". Just as below:
"ELK host ip" is my ELK host ip address.

#============================== Kibana =====================================
setup.kibana:
  host: "ELK host ip:5601"
#================================ Outputs =====================================
# Configure what output to use when sending the data collected by the beat.
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  # Array of hosts to connect to.
  hosts: ["ELK host ip:9200"]

Then, here is my "filebeat.yml" input.

#========== Filebeat inputs ==========
filebeat.inputs:
- type: log
  enabled: true
  paths:
    - C:\testcase\*.log

I had tried the command ".\filebeat test config .\filebeat.yml" and get the result of "Config OK".
Then, I make some testing logs in the path and "Start-Service Filebeat".

But, Kibana can't see any message about "Filebeat". How do I solve it?

[warkolm (Mark Walkom)]

Please don't post pictures of text, they are difficult to read, impossible to search and replicate (if it's code), and some people may not be even able to see them

What do the Filebeat logs show once it's started?

[stanwang]

I'm sorry about that.

Here is my Filebeat log.

2020-05-26T15:08:15.846+0800	INFO	instance/beat.go:621	Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\Program Files\Filebeat\data] Logs path: [C:\Program Files\Filebeat\logs]
2020-05-26T15:08:15.884+0800	INFO	instance/beat.go:629	Beat ID: b0535027-60e5-4d11-8178-2a678de322d0
2020-05-26T15:08:15.896+0800	INFO	[beat]	instance/beat.go:957	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Filebeat", "data": "C:\\Program Files\\Filebeat\\data", "home": "C:\\Program Files\\Filebeat", "logs": "C:\\Program Files\\Filebeat\\logs"}, "type": "filebeat", "uuid": "b0535027-60e5-4d11-8178-2a678de322d0"}}}
2020-05-26T15:08:15.896+0800	INFO	[beat]	instance/beat.go:966	Build info	{"system_info": {"build": {"commit": "5e69e25b920e3d93bec76a09a31da3ab35a55607", "libbeat": "7.7.0", "time": "2020-05-12T00:53:14.000Z", "version": "7.7.0"}}}
2020-05-26T15:08:15.896+0800	INFO	[beat]	instance/beat.go:969	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.13.9"}}}
2020-05-26T15:08:15.902+0800	INFO	[beat]	instance/beat.go:973	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-25T07:21:40.25+08:00","name":"Stan-VM","ip":["fe80::d90a:ca:694d:a307/64","[Client host ip]/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.836 (WinBuild.160101.0800)","mac":["00:0c:29:5b:ab:7c"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.836"},"timezone":"CST","timezone_offset_sec":28800,"id":"31552a7a-6e38-4d05-9b26-009a907b3320"}}}
2020-05-26T15:08:15.920+0800	INFO	[beat]	instance/beat.go:1002	Process info	{"system_info": {"process": {"cwd": "C:\\Program Files\\Filebeat", "exe": "C:\\Program Files\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 8912, "ppid": 7568, "start_time": "2020-05-26T15:08:15.218+0800"}}}
2020-05-26T15:08:15.920+0800	INFO	instance/beat.go:297	Setup Beat: filebeat; Version: 7.7.0
2020-05-26T15:08:15.920+0800	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'filebeat-7.7.0' as ILM is enabled.
2020-05-26T15:08:15.920+0800	INFO	eslegclient/connection.go:84	elasticsearch url: http://[ELK host ip]:9200
2020-05-26T15:08:15.920+0800	INFO	[publisher]	pipeline/module.go:110	Beat name: Stan-VM

[stanwang]

I had used the command on PowerShell :

.\filebeat.exe -c .\filebeat.yml -e -d "*"

It seems that Filebeat is work correctly on my local client. And, here is my Filebeat logs.
I put a empty log named ,"ttttttt.log".

2020-05-28T13:53:12.098+0800	INFO	instance/beat.go:622	Home path: [C:\Program Files\Filebeat] Config path: [C:\Program Files\Filebeat] Data path: [C:\ProgramData\filebeat] Logs path: [C:\ProgramData\filebeat\logs]
2020-05-28T13:53:12.128+0800	INFO	instance/beat.go:630	Beat ID: ae2d8b74-998f-4903-b9aa-12c3d6e32045
2020-05-28T13:53:12.140+0800	INFO	[beat]	instance/beat.go:958	Beat info	{"system_info": {"beat": {"path": {"config": "C:\\Program Files\\Filebeat", "data": "C:\\ProgramData\\filebeat", "home": "C:\\Program Files\\Filebeat", "logs": "C:\\ProgramData\\filebeat\\logs"}, "type": "filebeat", "uuid": "ae2d8b74-998f-4903-b9aa-12c3d6e32045"}}}
2020-05-28T13:53:12.140+0800	INFO	[beat]	instance/beat.go:967	Build info	{"system_info": {"build": {"commit": "d57bcf8684602e15000d65b75afcd110e2b12b59", "libbeat": "7.6.2", "time": "2020-03-26T05:23:36.000Z", "version": "7.6.2"}}}
2020-05-28T13:53:12.140+0800	INFO	[beat]	instance/beat.go:970	Go runtime info	{"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.13.8"}}}
2020-05-28T13:53:12.146+0800	INFO	[beat]	instance/beat.go:974	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-05-28T12:02:42.78+08:00","name":"Stan-VM","ip":["fe80::d90a:ca:694d:a307/64","10.3.87.201/16","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.836 (WinBuild.160101.0800)","mac":["00:0c:29:5b:ab:7c"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.836"},"timezone":"CST","timezone_offset_sec":28800,"id":"31552a7a-6e38-4d05-9b26-009a907b3320"}}}
2020-05-28T13:53:12.150+0800	INFO	[beat]	instance/beat.go:1003	Process info	{"system_info": {"process": {"cwd": "C:\\Windows\\system32", "exe": "C:\\Program Files\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 4232, "ppid": 680, "start_time": "2020-05-28T13:53:11.508+0800"}}}
2020-05-28T13:53:12.150+0800	INFO	instance/beat.go:298	Setup Beat: filebeat; Version: 7.6.2
2020-05-28T13:53:12.150+0800	INFO	[index-management]	idxmgmt/std.go:182	Set output.elasticsearch.index to 'filebeat-7.6.2' as ILM is enabled.
2020-05-28T13:53:12.151+0800	INFO	elasticsearch/client.go:174	Elasticsearch url: http://[ELK host ip]:9200
2020-05-28T13:53:12.151+0800	INFO	[publisher]	pipeline/module.go:110	Beat name: Stan-VM
2020-05-28T13:53:12.152+0800	INFO	instance/beat.go:439	filebeat start running.
2020-05-28T13:53:12.152+0800	INFO	[monitoring]	log/log.go:118	Starting metrics logging every 30s
2020-05-28T13:53:12.158+0800	INFO	registrar/registrar.go:145	Loading registrar data from C:\ProgramData\filebeat\registry\filebeat\data.json
2020-05-28T13:53:12.158+0800	INFO	registrar/registrar.go:152	States Loaded from registrar: 2
2020-05-28T13:53:12.158+0800	INFO	crawler/crawler.go:72	Loading Inputs: 1
2020-05-28T13:53:12.158+0800	INFO	log/input.go:152	Configured paths: [C:\Users\Stan_VM_Win10_1\Desktop\testcase\*]
2020-05-28T13:53:12.158+0800	INFO	input/input.go:114	Starting input of type: log; ID: 10865679355279554090 
2020-05-28T13:53:12.158+0800	INFO	crawler/crawler.go:106	Loading and starting Inputs completed. Enabled inputs: 1
2020-05-28T13:53:12.160+0800	INFO	cfgfile/reload.go:175	Config reloader started
2020-05-28T13:53:12.160+0800	INFO	cfgfile/reload.go:235	Loading of config files completed.
2020-05-28T13:53:15.151+0800	INFO	add_cloud_metadata/add_cloud_metadata.go:89	add_cloud_metadata: hosting provider type not detected.
2020-05-28T13:53:42.154+0800	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78,"time":{"ms":78}},"total":{"ticks":109,"time":{"ms":109},"value":109},"user":{"ticks":31,"time":{"ms":31}}},"handles":{"open":249},"info":{"ephemeral_id":"aad0228c-23ec-408c-a7b4-fcd0e0f9deb0","uptime":{"ms":30103}},"memstats":{"gc_next":11883840,"memory_alloc":8270072,"memory_total":13680456,"rss":36716544},"runtime":{"goroutines":25}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":2,"update":1},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":4}}}}}
2020-05-28T13:54:12.154+0800	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":78},"total":{"ticks":109,"value":109},"user":{"ticks":31}},"handles":{"open":247},"info":{"ephemeral_id":"aad0228c-23ec-408c-a7b4-fcd0e0f9deb0","uptime":{"ms":60104}},"memstats":{"gc_next":11883840,"memory_alloc":8353680,"memory_total":13764064,"rss":36864},"runtime":{"goroutines":25}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":2}}}}}
2020-05-28T13:54:22.166+0800	INFO	log/harvester.go:297	Harvester started for file: C:\Users\Stan_VM_Win10_1\Desktop\testcase\ttttttt.log
2020-05-28T13:54:42.153+0800	INFO	[monitoring]	log/log.go:145	Non-zero metrics in the last 30s	{"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":93,"time":{"ms":15}},"total":{"ticks":124,"time":{"ms":15},"value":124},"user":{"ticks":31}},"handles":{"open":248},"info":{"ephemeral_id":"aad0228c-23ec-408c-a7b4-fcd0e0f9deb0","uptime":{"ms":90103}},"memstats":{"gc_next":11883840,"memory_alloc":8466784,"memory_total":13877168,"rss":90112},"runtime":{"goroutines":30}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"files":{"72369424-3364-40d4-a720-c3c23f0b0cad":{"last_event_published_time":"","last_event_timestamp":"","name":"C:\\Users\\Stan_VM_Win10_1\\Desktop\\testcase\\ttttttt.log","start_time":"2020-05-28T13:54:22.165Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":3,"update":1},"writes":{"success":1,"total":1}}}}}

How do I sent this message to the Elasticsearch?