OK Fixed 
Now its time to SSL the connection to the browser
So now I have access to Kibana again (fresh install OS and stack) but Im getting the same message.
[2024-05-10T13:58:51,929][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37272}
[2024-05-10T13:58:51,931][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37276}
[2024-05-10T13:58:51,944][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37282}
[2024-05-10T13:58:51,944][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37284}
[2024-05-10T13:58:51,946][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37322}
[2024-05-10T13:58:51,946][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37324}
[2024-05-10T13:58:51,946][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37298}
[2024-05-10T13:58:51,947][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37310}
[2024-05-10T13:58:51,948][WARN ][o.e.h.n.Netty4HttpServerTransport] [servidor] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:37332}
[2024-05-10T13:58:59,792][WARN ][o.e.h.AbstractHttpServerTransport] [servidor] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/172.20.5.100:9200, remoteAddress=/172.22.40.11:61378}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
... 16 more
Those connections are not coming from elasticsearch or Kibana as far as I can tell.
And a number of them appear to be just as the message says something trying to connect to http on the https interface and then the last one is trying to do some sort of certificate which is invalid.
I am not sure if you have sanitized the IP addresses etc or not.
Is this server on a private Network? Is it exposed to the internet?
I have seen on occasion internal corporate security scanners do something similar.
But I really can't tell you where those are coming from. They're not coming from Elasticsearch or Kibana as far as I can tell
By the way, I can explain this if you are interested. In short, a setting is put in the keystore when you do auto setup and if you just remove all the SSL settings it's still there so the configuration is not consistent.
Those connections are not coming from elasticsearch or Kibana as far as I can tell.
After DEEP investigation, they are from the Elastic Agent installed on my PC. I fixed and now the only error I have is:
[2024-05-13T09:39:12,770][WARN ][o.e.h.AbstractHttpServerTransport] [server] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:45144}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
... 16 more
Is this server on a private Network? Is it exposed to the internet?
Everything is on a private network. Nothing is exposed (nor will be)
Hi @riahc3
So when you say you have fixed now, What does that mean?
Do you have https on?
Transport HTTPS
Elastic HTTPS
Kibana HTTPS
And everything is working except for that error message?
Those error messages. Are they repeating or is it a single in time?.
According to the error message, there is something coming from 127.0.0.1 ie localhost trying to connect and presenting a bad certificate.
I've seen this with some sorts of corporate scanners that are installed on the os's.
But I don't have any clue what that is....
I think you should be able to use lsof to figure out what process that connection is originating from.
Or You could turn up the logs to trace and see if you get any more information...
So when you say you have fixed now, What does that mean?
What I mean is that the only error message that appears is this:
[2024-05-13T09:39:12,770][WARN ][o.e.h.AbstractHttpServerTransport] [server] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=/127.0.0.1:9200, remoteAddress=/127.0.0.1:45144}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:499) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:290) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:444) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:412) ~[?:?]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:440) ~[?:?]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:420) ~[?:?]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) ~[?:?]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:166) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:788) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:689) ~[?:?]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:652) ~[?:?]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:562) ~[?:?]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997) ~[?:?]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) ~[?:?]
at java.lang.Thread.run(Thread.java:1583) ~[?:?]
Caused by: javax.net.ssl.SSLHandshakeException: Received fatal alert: bad_certificate
at sun.security.ssl.Alert.createSSLException(Alert.java:130) ~[?:?]
at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]
at sun.security.ssl.TransportContext.fatal(TransportContext.java:365) ~[?:?]
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:287) ~[?:?]
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:204) ~[?:?]
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:172) ~[?:?]
at sun.security.ssl.SSLEngineImpl.decode(SSLEngineImpl.java:736) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:691) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:506) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:482) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:679) ~[?:?]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:297) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1353) ~[?:?]
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1246) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1295) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:529) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:468) ~[?:?]
... 16 more
I imagine this is from my PC's Elastic Agent to Elasticsearch....
The only thing that is missing right now is when I access Kibana on 5601, the SSL part.
Those error messages. Are they repeating or is it a single in time?.
It repeats
I've seen this with some sorts of corporate scanners that are installed on the os's.
There isnt anything installed on this Debian box.
I think you should be able to use lsof to figure out what process that connection is originating from.
Here are the results:
root@server:~# lsof -i tcp:9200
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
node 734 kibana 18u IPv4 326540 0t0 TCP 172.20.5.100:48394->172.20.5.100:9200 (ESTABLISHED)
node 734 kibana 21u IPv4 328723 0t0 TCP 172.20.5.100:58750->172.20.5.100:9200 (ESTABLISHED)
node 734 kibana 22u IPv4 326541 0t0 TCP 172.20.5.100:48410->172.20.5.100:9200 (ESTABLISHED)
node 734 kibana 101u IPv4 311806 0t0 TCP 172.20.5.100:46736->172.20.5.100:9200 (ESTABLISHED)
java 14147 elasticsearch 43u IPv4 327739 0t0 TCP 172.20.5.100:9200->172.20.5.100:58750 (ESTABLISHED)
java 14147 elasticsearch 451u IPv4 297928 0t0 TCP *:9200 (LISTEN)
java 14147 elasticsearch 467u IPv4 325606 0t0 TCP 172.20.5.100:9200->172.22.40.11:7226 (ESTABLISHED)
java 14147 elasticsearch 574u IPv4 326543 0t0 TCP 172.20.5.100:9200->172.22.40.11:9108 (ESTABLISHED)
java 14147 elasticsearch 702u IPv4 328058 0t0 TCP 172.20.5.100:9200->172.20.5.100:48394 (ESTABLISHED)
java 14147 elasticsearch 709u IPv4 328060 0t0 TCP 172.20.5.100:9200->172.20.5.100:48410 (ESTABLISHED)
java 14147 elasticsearch 795u IPv4 314500 0t0 TCP 172.20.5.100:9200->172.20.5.100:46736 (ESTABLISHED)
It seems to be Kibana connecting to Elasticsearch.
perhaps.... or fleet did you install a fleet server?
I don't think so those connections are Established... and Kibana only use the connection information in the kibana.yml either all work or none work
I suspect it is the agent or something else...
you may need to run lsof a few time to catch it because elasticsearch is terminating the connection...
You might need to run lsof with closed_wait to see what it is...
perhaps.... or fleet did you install a fleet server?
No, I didnt install a Fleet Server on my Windows (and my linux installation is fresh)
I suspect it is the agent or something else...
I went ahead and uninstalled the agent on my Windows PC; The linux installation is brand new so it shouldnt have the agent or anything installed AFAIK
I ran lsof -i several times and I dont see a close_wait anywhere
Could a tcpdump give us more information on the process starting the connection?
Only Elasticsearch, Kibana, Metricbeat and Logstash are installed on this server.
metricbeat or logstash could be trying to connect if they are running.... First you mentioned these
Are they running? Are they properly configured?
.
metricbeat or logstash could be trying to connect if they are running.... First you mentioned these :slight_smile:
My mistake: I thought Logstash was a givein as this is a ELK stack and Metricbeat I needed for self cluster monitoring.
Are they running? Are they properly configured?
They SEEM to be running....
Looking at Logstash logs, everything seems to be in order.
There is nothing given when trying to debug something,... Especially remotely... Through a forum like this... each component can be installed separately... Many people use ELK as a very generic term...
Just because they're running doesn't mean they're connected properly... Doesn't mean they're not the source of your SSL problem.... I'm not saying they are but they certainly could be.
I think you just need to dig around
If you want to share those configs happy to take a look at them.
If you want to share those configs happy to take a look at them.
Sure I can send them but obviously all would be thru private messages.
Which ones do you need?
Thank you for all your help
Apologies, I / We do not answer topics through private messages...
Perhaps simply stop logstash and metricbeat and see if the problems persist.
Not sure what to tell you... think you are going to need to just continue to debug.
More of "a problem", right now, the issue is correctly securing the connection when I connect to Kibana: http://ipaddressofkibana:5601 to https://ipaddressofkibana:5601
(I have another issue with integration but thats another seperate)
The cert for that HTTPS for Kibana is just a normal cert... So however you create it to work with your company security process will work.
I have created self-signed, I have created publicly signed like let's encrypt... We have many users that have created using their company's CSR method... That's up to you. It's just a cert, Kibana does nothing special in that world for that connection. It's just like any other web app you would run in your company.
The cert for that HTTPS for Kibana is just a normal cert... So however you create it to work with your company security process will work.
Thats what I thought too
The issue is that when I tried to set it up everything broke so....Im not sure what in the config is required to set it up.
I perfer to set it up with my company´s CA