Applying SSL to Elasticsearch and Kibana: Connection refused on Kibana

Elastic Stack Elasticsearch
41

OK Fixed :slight_smile:

Now its time to SSL the connection to the browser

42

So now I have access to Kibana again (fresh install OS and stack) but Im getting the same message.

[2024-05-10T13:58:51,929][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37272}
[2024-05-10T13:58:51,931][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37276}
[2024-05-10T13:58:51,944][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37282}
[2024-05-10T13:58:51,944][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37284}
[2024-05-10T13:58:51,946][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37322}
[2024-05-10T13:58:51,946][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37324}
[2024-05-10T13:58:51,946][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37298}
[2024-05-10T13:58:51,947][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37310}
[2024-05-10T13:58:51,948][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37332}
[2024-05-10T13:58:59,792][WARN ][o.e.h.AbstractHttpServerTransport] [servidor] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.20.5.100:9200, remoteAddress=/172.22.40.11:61378}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more
43

Those connections are not coming from elasticsearch or Kibana as far as I can tell.

And a number of them appear to be just as the message says something trying to connect to http on the https interface and then the last one is trying to do some sort of certificate which is invalid.

I am not sure if you have sanitized the IP addresses etc or not.

Is this server on a private Network? Is it exposed to the internet?

I have seen on occasion internal corporate security scanners do something similar.

But I really can't tell you where those are coming from. They're not coming from Elasticsearch or Kibana as far as I can tell

By the way, I can explain this if you are interested. In short, a setting is put in the keystore when you do auto setup and if you just remove all the SSL settings it's still there so the configuration is not consistent.

44

Those connections are not coming from elasticsearch or Kibana as far as I can tell.

After DEEP investigation, they are from the Elastic Agent installed on my PC. I fixed and now the only error I have is:

[2024-05-13T09:39:12,770][WARN ][o.e.h.AbstractHttpServerTransport] [server] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:45144}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more

Is this server on a private Network? Is it exposed to the internet?

Everything is on a private network. Nothing is exposed (nor will be)

45

Hi @riahc3

So when you say you have fixed now, What does that mean?

Do you have https on?
Transport HTTPS
Elastic HTTPS
Kibana HTTPS

And everything is working except for that error message?

Those error messages. Are they repeating or is it a single in time?.

According to the error message, there is something coming from 127.0.0.1 ie localhost trying to connect and presenting a bad certificate.

I've seen this with some sorts of corporate scanners that are installed on the os's.

But I don't have any clue what that is....

I think you should be able to use lsof to figure out what process that connection is originating from.

Or You could turn up the logs to trace and see if you get any more information...

46

So when you say you have fixed now, What does that mean?

What I mean is that the only error message that appears is this:


[2024-05-13T09:39:12,770][WARN ][o.e.h.AbstractHttpServerTransport] [server] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:45144}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
        at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
        at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
        at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
        at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
        at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
        at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
        at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
        at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
        at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
        at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?]
        at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
        at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
        at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
        at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
        at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
        at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[?:?]
        at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
        at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
        at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
        ... 16 more

I imagine this is from my PC's Elastic Agent to Elasticsearch....

The only thing that is missing right now is when I access Kibana on 5601, the SSL part.

Those error messages. Are they repeating or is it a single in time?.

It repeats

I've seen this with some sorts of corporate scanners that are installed on the os's.

There isnt anything installed on this Debian box.

I think you should be able to use lsof to figure out what process that connection is originating from.

Here are the results:

root@server:~# lsof -i tcp:9200
COMMAND   PID          USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
node      734        kibana   18u  IPv4 326540      0t0  TCP 172.20.5.100:48394->172.20.5.100:9200 (ESTABLISHED)
node      734        kibana   21u  IPv4 328723      0t0  TCP 172.20.5.100:58750->172.20.5.100:9200 (ESTABLISHED)
node      734        kibana   22u  IPv4 326541      0t0  TCP 172.20.5.100:48410->172.20.5.100:9200 (ESTABLISHED)
node      734        kibana  101u  IPv4 311806      0t0  TCP 172.20.5.100:46736->172.20.5.100:9200 (ESTABLISHED)
java    14147 elasticsearch   43u  IPv4 327739      0t0  TCP 172.20.5.100:9200->172.20.5.100:58750 (ESTABLISHED)
java    14147 elasticsearch  451u  IPv4 297928      0t0  TCP *:9200 (LISTEN)
java    14147 elasticsearch  467u  IPv4 325606      0t0  TCP 172.20.5.100:9200->172.22.40.11:7226 (ESTABLISHED)
java    14147 elasticsearch  574u  IPv4 326543      0t0  TCP 172.20.5.100:9200->172.22.40.11:9108 (ESTABLISHED)
java    14147 elasticsearch  702u  IPv4 328058      0t0  TCP 172.20.5.100:9200->172.20.5.100:48394 (ESTABLISHED)
java    14147 elasticsearch  709u  IPv4 328060      0t0  TCP 172.20.5.100:9200->172.20.5.100:48410 (ESTABLISHED)
java    14147 elasticsearch  795u  IPv4 314500      0t0  TCP 172.20.5.100:9200->172.20.5.100:46736 (ESTABLISHED)

It seems to be Kibana connecting to Elasticsearch.

47

perhaps.... or fleet did you install a fleet server?

I don't think so those connections are Established... and Kibana only use the connection information in the kibana.yml either all work or none work

I suspect it is the agent or something else...

you may need to run lsof a few time to catch it because elasticsearch is terminating the connection...

You might need to run lsof with closed_wait to see what it is...

48

perhaps.... or fleet did you install a fleet server?

No, I didnt install a Fleet Server on my Windows (and my linux installation is fresh)

I suspect it is the agent or something else...

I went ahead and uninstalled the agent on my Windows PC; The linux installation is brand new so it shouldnt have the agent or anything installed AFAIK

I ran lsof -i several times and I dont see a close_wait anywhere

Could a tcpdump give us more information on the process starting the connection?

Only Elasticsearch, Kibana, Metricbeat and Logstash are installed on this server.

49

metricbeat or logstash could be trying to connect if they are running.... First you mentioned these :slight_smile:

Are they running? Are they properly configured?

.

50

metricbeat or logstash could be trying to connect if they are running.... First you mentioned these :slight_smile:

My mistake: I thought Logstash was a givein as this is a ELK stack and Metricbeat I needed for self cluster monitoring.

Are they running? Are they properly configured?

image
image1400×239 10.8 KB

image
image1903×385 17.4 KB

They SEEM to be running....

Looking at Logstash logs, everything seems to be in order.

51

There is nothing given when trying to debug something,... Especially remotely... Through a forum like this... each component can be installed separately... Many people use ELK as a very generic term...

Just because they're running doesn't mean they're connected properly... Doesn't mean they're not the source of your SSL problem.... I'm not saying they are but they certainly could be.

I think you just need to dig around

If you want to share those configs happy to take a look at them.

52

If you want to share those configs happy to take a look at them.

Sure I can send them but obviously all would be thru private messages.

Which ones do you need?

Thank you for all your help

53

Apologies, I / We do not answer topics through private messages...

Perhaps simply stop logstash and metricbeat and see if the problems persist.

Not sure what to tell you... think you are going to need to just continue to debug.

54

More of "a problem", right now, the issue is correctly securing the connection when I connect to Kibana: http://ipaddressofkibana:5601 to https://ipaddressofkibana:5601

(I have another issue with integration but thats another seperate)

55

The cert for that HTTPS for Kibana is just a normal cert... So however you create it to work with your company security process will work.

I have created self-signed, I have created publicly signed like let's encrypt... We have many users that have created using their company's CSR method... That's up to you. It's just a cert, Kibana does nothing special in that world for that connection. It's just like any other web app you would run in your company.

56

The cert for that HTTPS for Kibana is just a normal cert... So however you create it to work with your company security process will work.

Thats what I thought too

The issue is that when I tried to set it up everything broke so....Im not sure what in the config is required to set it up.

I perfer to set it up with my company´s CA