Winlogbeat not working after enabling SSL but other beats work

Elastic Stack Beats
1

Hello there,

We are at ELK 6.4.2 (just upgraded) and beats are at 6.3.2.
I did try winlogbeat 6.4.2, the same error.

Recently we enabled X-Pack and set up SSL\TLS on Logstash/Elasticsearch/Kibana.
the Beats (filebeat/metricbeat/auditbeat/packetbeat) work fine on linux with the SSL (certs/keys) enabled on them. But on Windows, Packetbeat and Auditbeat work fine with SSL enabled. But winlogbeat can doesn't work, it seems that winlogbeat worked for 1 or 2 minutes then stopped..
Nothing was changed on the winlogbeat.yml except the output to Logstas with SSL certs/keys on.
Using the same SSL certs/keys on the same window server, auditbeat and packet beats work fine.

It seems it still looks for http instead of https.
"2018-11-01T14:09:56.401-0500 INFO kibana/client.go:90 Kibana url: http://kibanahost:5601"

Here is the testing output on Windlogbeat:

PS C:\Program Files\winlogbeat-6.3.2> .\winlogbeat -c winlogbeat.yml -e -v
2018-11-01T14:09:56.380-0500 INFO instance/beat.go:492 Home path: [C:\ProgramFiles\winlogbeat-6.3.2] Config path: [C:\Program Files\winlogbeat-6.3.2] Data path: [C:\Program Files\winlogbeat-6.3.2\data] Logs path: [C:\Program Files\winlogbeat-6.3.2\logs]
2018-11-01T14:09:56.383-0500 INFO instance/beat.go:499 Beat UUID: 71d473ab-95a7-4807-91c5-5fb62c0fb510
2018-11-01T14:09:56.383-0500 INFO [beat] instance/beat.go:716 Beat info {"system_info": {"beat": {"path": {"config": "C:\Program Files\winlogbeat-6.3.2", "data": "C:\Program Files\winlogbeat-6.3.2\data", "home": "C:\Program Files\winlogbeat-6.3.2", "logs": "C:\Program Files\winlogbeat-6.3.2\logs"}, "type": "winlogbeat", "uuid": "71d473ab-95a7-4807-91c5-5fb62c0fb510"}}}
2018-11-01T14:09:56.385-0500 INFO [beat] instance/beat.go:725 Build info {"system_info": {"build": {"commit": "45a9a9e1561b6c540e94211ebe03d18abcacae55", "libbeat": "6.3.2", "time": "2018-07-20T04:21:29.000Z", "version": "6.3.2"}}}
2018-11-01T14:09:56.386-0500 INFO [beat] instance/beat.go:728 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":4,"version":"go1.9.4"}}}
2018-11-01T14:09:56.393-0500 INFO [beat] instance/beat.go:732 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2018-10-31T08:51:56.35-05:00","hostname":"hlsoadse1a-01","ips":["fe80::d426:c557:6fb9:d34
6/64","10.100.16.11/23","::1/128","127.0.0.1/8","fe80::5efe:a64:100b/128"],"kernel_version":"6.3.9600.19153 (winblue_ltsb.180908-0600)","mac_addresses":["0e:ec:25:c3:39:8c","00:00:00:00:00:00:00:e0"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19156"},"timezone":"CDT","timezone_offset_sec":-18000,"id":"5da79a47-449b-45f7-8e90-be2395659d97"}}}
2018-11-01T14:09:56.395-0500 INFO instance/beat.go:225 Setup Beat: winlogbeat; Version: 6.3.2
2018-11-01T14:09:56.399-0500 INFO pipeline/module.go:81 Beat name: beathost-01
2018-11-01T14:09:56.400-0500 INFO beater/winlogbeat.go:51 State will be read from and persisted to C:\Program Files\winlogbeat-6.3.2\data.winlogbeat.yml
2018-11-01T14:09:56.401-0500 INFO [monitoring] log/log.go:97 Starting metrics logging every 30s
2018-11-01T14:09:56.401-0500 INFO kibana/client.go:90 Kibana url: http://kibanahost:5601
2018-11-01T14:10:17.540-0500 INFO [monitoring] log/log.go:132 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":109,"time":{"ms":109}},"total":{"ticks":202,"time":{"ms":202},"value":202},"user":{
"ticks":93,"time":{"ms":93}}},"info":{"ephemeral_id":"51777cb8-38d4-4ca4-9701-ee7cc0bc0382","uptime":{"ms":26108}},"memstats":{"gc_next":4194304,"memory_alloc":1550432,"memory_total":3282568,"rss":17772544}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":4}}}}}
2018-11-01T14:10:17.543-0500 INFO [monitoring] log/log.go:133 Uptime: 26.2350229s
2018-11-01T14:10:17.543-0500 INFO [monitoring] log/log.go:110 Stopping metrics logging.
2018-11-01T14:10:17.543-0500 INFO instance/beat.go:306 winlogbeat stopped.
2018-11-01T14:10:17.544-0500 ERROR instance/beat.go:691 Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get http://hlsokbne1a-01.hls.dxc.com:5601/api/status: dial tcp 10.100.35.15:5601: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.. Response: .

Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET re
quest: Get http://kibanahost-01.hls.dxc.com:5601/api/status: dial tcp 10.100.35.15:5601: connectex: A connection attempt
failed because the connected party did not properly respond after a period of time, or established connection failed be
cause connected host has failed to respond.. Response: .
PS C:\Program Files\winlogbeat-6.3.2>

PS C:\Program Files\winlogbeat-6.3.2> .\winlogbeat.exe test config -c .\winlogbeat.yml -e
...
Config OK

PS C:\Program Files\winlogbeat-6.3.2> .\winlogbeat.exe test output -e -d "*"
2018-11-01T14:56:30.556-0500 INFO instance/beat.go:492 Home path: [C:\Program Files\winlogbeat-6.3.2] Config path: [C:\Program Files\winlogbeat-6.3.2] Data path: [C:\Program Files\winlogbeat-6.3.2\data] Logs path: [C:\Program Files\winlogbeat-6.3.2\logs]
2018-11-01T14:56:30.558-0500 DEBUG [beat] instance/beat.go:519 Beat metadata path: C:\Program Files\winlogbeat-6.3.2\data\meta.json
2018-11-01T14:56:30.559-0500 INFO instance/beat.go:499 Beat UUID: 71d473ab-95a7-4807-91c5-5fb62c0fb510
logstash: logstashhost:5044...
connection...
parse host... OK
dns lookup... OK
addresses: 10.100.35.15
dial up... OK
TLS...
security: server's certificate chain verification is enabled

==================================

Again, auditbeat and packetbeat on the same windows server work fine...

Windows server is Windows2012R2 64 bit.

I didn't post the winlogbeat.yml because nothing changed on it except the output to logstash portion with SSL enabled and the same setting (I copied over from auditbeat) on auditbeat/packetbeat work fine.

Please take a look and help...

Thanks in advance

Li

2

I deleted the registry file, .winlogbeat.yml, still the same error...

3

Any updates on this, please?

4

It looks like you're experience a potentially flaky connection when I see this error:

Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET re
quest: Get http://kibanahost-01.hls.dxc.com:5601/api/status: dial tcp 10.100.35.15:5601: connectex: A connection attempt
failed because the connected party did not properly respond after a period of time, or established connection failed be
cause connected host has failed to respond.. Response: .
PS C:\Program Files\winlogbeat-6.3.2>

Can you test the connectivity between your windows box and Kibana. Is it stable? Is kibana reliably reachable from there?

5

Auditbeat and Packetbeat on the same windows server work fine....
They use the exactly the same certs and keys...

here is the log, it showed Winlogbeat stopped before the ERROR entry...Please read log items below...

---------- 2018-11-01T14:10:17.543-0500 INFO instance/beat.go:306 winlogbeat stopped.
---------- 2018-11-01T14:10:17.544-0500 ERROR instan

"ticks":93,"time":{"ms":93}}},"info":{"ephemeral_id":"51777cb8-38d4-4ca4-9701-ee7cc0bc0382","uptime":{"ms":26108}},"memstats":{"gc_next":4194304,"memory_alloc":1550432,"memory_total":3282568,"rss":17772544}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"system":{"cpu":{"cores":4}}}}}
2018-11-01T14:10:17.543-0500 INFO [monitoring] log/log.go:133 Uptime: 26.2350229s
2018-11-01T14:10:17.543-0500 INFO [monitoring] log/log.go:110 Stopping metrics logging.
2018-11-01T14:10:17.543-0500 INFO instance/beat.go:306 winlogbeat stopped.
2018-11-01T14:10:17.544-0500 ERROR instance/beat.go:691 Exiting: Error importing Kibana dashboards: fail to create the Kibana loader: Error creating Kibana client: Error creating Kibana client: fail to get the Kibana version: HTTP GET request to /api/status fails: fail to execute the HTTP GET request: Get http://hlsokbne1a-01.hls.dxc.com:5601/api/status: dial tcp 10.100.35.15:5601: connectex: A connection attempt failed because the connected party did not properly respond after a period of time, or established connection failed because connected host has failed to respond.. Response: .

6

Here is the winlogbeat.yml that didn't work, but all other beats (auditbeat, metricbeat, and packetbeat work on the same windows server with the same SSL related configuration):

setup.kibana:

Kibana Host

Scheme and port can be left out and will be set to the default (http and 5601)

In case you specify and additional path, the scheme is required: http://localhost:5601/path

IPv6 addresses should always be defined as: https://[2001:db8::1]:5601

#host: 'localhost:5601'
host: "https://kibanahost:5601"
username: "kibana"
password: "pass"

...
#----------------------------- Logstash output --------------------------------
output.logstash:

The Logstash hosts

#hosts: ['localhost:5044']
hosts: ['logstashhost.com:5044']

Optional SSL. By default is off.

List of root certificates for HTTPS server verifications

ssl.certificate_authorities: ['C:\Windows\keys\entrust_g2.crt','C:\Windows\keys\entrust_l1k.crt','C:\Windows\keys\entrust_rootca1.crt']
#ssl.certificate_authorities: ['/etc/pki/root/ca.pem']
ssl.certificate: 'C:\Windows\keys\ServerCertificate.crt'

Certificate for SSL client authentication

#ssl.certificate: '/etc/pki/client/cert.pem'

Client Certificate Key

#ssl.key: '/etc/pki/client/cert.key'
ssl.key: 'C:\Windows\keys\hls-201710-mykey.pem'

All beats, use the same SSL configuration and we have X-Pack enabled on Logstash/Elastic../kibana as well.

Thanks a lot

7

Any updates on this, please?

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.