Winlogbeat can't connect to the ELK Stack server

aboutELK · March 10, 2020, 1:13pm

I created ELK Stack Server with Ubuntu 18.04 LTS and a Linux client with filebeat. Everything works fine together. I tried to install winlogbeat on my Windows PCs. But I can't get it working together. The error log from winlogbeat shows the following messages. I would really appreciate if someone comes forward to help me.

The german error message means s.th. like "No connection possible because destination computer denies connection"

2020-03-09T14:50:25.017+0100 ERROR elasticsearch/elasticsearch.go:261 Error connecting to Elasticsearch at http://192.168.189.110:9200: Get http://192.168.189.110:9200: dial tcp 192.168.189.110:9200: connectex: Es konnte keine Verbindung hergestellt werden, da der Zielcomputer die Verbindung verweigerte.
2020-03-09T14:50:25.017+0100 ERROR instance/beat.go:933 Exiting: Couldn't connect to any of the configured Elasticsearch hosts. Errors: [Error connection to Elasticsearch http://192.168.189.110:9200: Get http://192.168.189.110:9200: dial tcp 192.168.189.110:9200: connectex: Es konnte keine Verbindung hergestellt werden, da der Zielcomputer die Verbindung verweigerte.]

rugenl · March 10, 2020, 1:37pm

It could be a firewall block. You can test with

nmap -p 9200 192.168.189.110

Run this on the system with the filebeat client.

Edit: I missed that you had both filebeat and winlogbeat in the post. The winlogbeat problem could be firewall, but you will have to test with a windows tool from the failing windows hosts. Some here use "telnet host port" from windows, but I don't do windows

aboutELK · March 10, 2020, 2:42pm

we could solve this issue but now have another one:
We are trying to send the data from winlogbeat to logstash on our elk server. From another linux server filebeat is working. We suppose it is an issue with the certificates but cannot get it working.
Any help is appreciated.

2020-03-10T15:35:36.447+0100 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":984},"total":{"ticks":153843,"time":{"ms":3234},"value":153843},"user":{"ticks":152859,"time":{"ms":3234}}},"handles":{"open":347},"info":{"ephemeral_id":"db6f6648-96c8-4b01-b3dd-083641d778ac","uptime":{"ms":1590274}},"memstats":{"gc_next":63315280,"memory_alloc":40786440,"memory_total":4159355432,"rss":49152},"runtime":{"goroutines":37}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"batches":38,"failed":77824,"total":77824},"read":{"bytes":114},"write":{"bytes":8334146}},"pipeline":{"clients":5,"events":{"active":4120,"retry":116736}}}}}}
2020-03-10T15:25:04.818+0100 INFO pipeline/output.go:95 Connecting to backoff(async(tcp://192.168.189.110:5443))
2020-03-10T15:25:04.820+0100 INFO pipeline/output.go:105 Connection to backoff(async(tcp://192.168.189.110:5443)) established
2020-03-10T15:25:04.927+0100 ERROR logstash/async.go:256 Failed to publish events caused by: lumberjack protocol error
2020-03-10T15:25:04.997+0100 ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected
2020-03-10T15:25:05.999+0100 ERROR pipeline/output.go:121 Failed to publish events: client is not connected

Counterpart on the elk server is:

Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: Mar 10 15:37:13 my-elk01 logstash[12101]: lines 944-1001/1001 (END) [2020-03-10T14:37:13,499][INFO ][org.logstash.beats.BeatsHandler][main] [local: 0.0.0.0:5443, remote: 192.168.180.219:57864] Handling e
[2020-03-10T14:37:13,507][WARN ][io.netty.channel.DefaultChannelPipeline][main] An exceptionCaught() event was fired, and it reached at
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSIO
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:472) ~[netty-all-4.1.30.Final.jar:4.1.30.Fi
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:278) ~[netty-all-4.1.30.Final.jar:4.1.30.F
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) ~[netty-all-4.1.30.Fi
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1434) ~[netty-all-4.1.30.Final.j
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) ~[netty-all-4.1.30.
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) ~[netty-all-4.1.30.
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:965) ~[netty-all-4.1.30.Final.jar:4.1.30
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) ~[netty-all-4.1.30.Final.jar
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:579) ~[netty-all-4.1.30.Final.jar:4.1.30.Fi
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:496) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897) [netty-all-4.1.30.Final.jar:4.1
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.30.Final.jar:4.1.30.Fin
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_242]
Caused by: javax.net.ssl.SSLHandshakeException: error:100000f7:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.sslReadErrorResult(ReferenceCountedOpenSslEngine.java:1140) ~[netty-all-4
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1101) ~[netty-all-4.1.30.Final.
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1169) ~[netty-all-4.1.30.Final.
at io.netty.handler.ssl.ReferenceCountedOpenSslEngine.unwrap(ReferenceCountedOpenSslEngine.java:1212) ~[netty-all-4.1.30.Final.
at io.netty.handler.ssl.SslHandler$SslEngineType$1.unwrap(SslHandler.java:216) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1297) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decodeNonJdkCompatible(SslHandler.java:1211) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1245) ~[netty-all-4.1.30.Final.jar:4.1.30.Final]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:502) ~[netty-all-4.1.30
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:441) ~[netty-all-4.1.30.Final.jar:4.1.30.Fi
... 16 more

rugenl · March 10, 2020, 6:54pm

What versions are involved?

aboutELK · March 11, 2020, 8:27am

We use this version:

Ubuntu 18.04.4 LTS
Elasticsearch 7.6.1
Kibana 7.6.1
Logstash 7.6.1-1
Winlogbeat 7.6.0 built 2020-02-05 23:15:28 +0000 UTC]

rugenl · March 11, 2020, 11:24pm

My guess is that winlogbeat isn't properly configured to attempt ssl encryption.

Can you post your configuration for logstash inputs, winlogbeat and just for comparison the working Linux filebeat?

aboutELK · March 12, 2020, 10:55am

For sure. This is the configuration for logstash inputs.

input {
beats {
port => 5443
type => syslog
ssl => true
ssl_certificate => "/etc/logstash/ssl/logstash-forwarder.crt"
ssl_key => "/etc/logstash/ssl/logstash-forwarder.key"
}
}

The configuration for Winlogbeat

winlogbeat.event_logs:

name: Application
ignore_older: 72h
name: System
name: Security

#----------------------------- Logstash output --------------------------------
output.logstash:
#The Logstash hosts
hosts: ["192.168.189.110:5443"]
index: "Winlogbeat"
#Optional SSL. By default is off.
#List of root certificates for HTTPS server verifications
#ssl.certificate_authorities: ["C:/ProgramData/Winlogbeat/logstash-forwarder.crt"]
#Certificate for SSL client authentication
#ssl.certificate: "c:/ProgramData/Winlogbeat/logstash-forwarder.crt"
#Client Certificate Key
#ssl.key: "/etc/pki/client/cert.key"

We're not sure where we can put ssl certificate for winlogbeat.
For Filebeat we use Ubuntu 18.04 LTS work fine with the ELK server.

rugenl · March 12, 2020, 12:40pm

Look here, but it's not very clear how it works with modules, You must have an "output.logstash hosts" somewhere in winlogbeat, try adding these ssl options in the same area.

Look at winlogbeat.reference.yml, it should have all possible options for all sections. Adding any SSL option will enable the output to attempt SSL connections.

system · April 9, 2020, 12:40pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.