Need Help with Installing Certificates into Elasticsearch

Elastic Stack Elasticsearch
1

I've been trying to get some certificates created by certgen installed but I keep encountering this error. I've tried many things like, but I'm at a brick wall. Any help would be much appreciated

Error:

Blockquote
Caused by: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[?:?]
at > io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

My problem is probably here but this is my first time doing certificates. I'm trying to get a cert installed for my localhost:9200. Instances.yml at ~/git/kibana_system/elasticsearch-5.5.2/

instances:

  • name: "node1"
    ip:
    • "127.0.0.1"

My certgen command:

sudo ./bin/x-pack/certgen -in /home/xxx/git/kibana_system/elasticsearch-5.5.2/instances.yml

My elasticsearch.yml

xpack.ssl.key: /home/xxx/git/kibana_system/elasticsearch-5.5.2/config/x-pack/alexnode/alexnode.key
xpack.ssl.certificate: /home/xxx/git/kibana_system/elasticsearch-5.5.2/config/x-pack/alexnode/alexnode.crt
xpack.ssl.certificate_authorities: [ "/home/xxx/git/kibana_system/elasticsearch-5.5.2/config/x-pack/ca/ca.crt" ]
xpack.ssl.verification_mode: full
xpack.security.transport.ssl.enabled: true
xpack.security.http.ssl.enabled: true

Any help would be appreciated! I got it working once, but I can't seem to figure out again. Been at this wall for two days now

2 
Caused by: javax.net.ssl.SSLException: Received fatal alert: bad_certificate

When does this error occur? At startup, or when you try and access the server?

From the stack trace, it looks like it's happening when 2 nodes try and come together to form a cluster.
Are you running multiple nodes? Or do you accidentally have another ES instance running on your machine?

Otherwise, nothing seems obviously wrong with your configuration.

3

Hi Tim,

I recieve this when I try to access my server.

I only have one node.

Where am I trying to add two?

I'm pretty sure that I only have one Elasticsearch instance going. If I add try to start a second instance on the same port it won't start.

Sorry for the late reply.

Thanks,

Alex

4

Can you provide more details about how you are accessing the server:

  • Are you using HTTP (REST) or transport client?
  • Are you seeing that error in the logs on the server, on in your client?

Where am I trying to add two?

The format of the stack trace with the ~[?:?] indicates that it is a serialized exception that has been sent from one JVM to another. Typically that happens when there are multiple nodes in a cluster and there is a failure that spans 2 (or more) nodes. Based on your most recent post, I now suspect this is a transport client error, rather than 2 nodes.

Which brings me to this request:
It's really helpful if you can provide as much useful information as you can in your posts.
The time I spend solving problems here is time that I don't have available to build features and fix bugs, and I am able to resolve issues more efficiently when I have all the information I need up front.
Things that I'm looking for are:

  • the version of elasticsearch & x-pack
  • whether errors occur on startup, on connection, or when processing requests.
  • whether errors are in the elasticsearch logs, or reported in the client.
  • which client you are using
  • what requests you are running

We'll always do what we can to work through issues, no matter how much (or little) information is provided, but you'll get a faster resolution if you can provide the key details at the very beginning.

5

Elasticsearch 5.5.2
X-pack 5.5.2

I start elasticsearch with ./bin/elasticsearch
Then I open my browser and try to access https://localhost:9200
Then I see this in my terminal window

[2017-10-31T10:45:31,021][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [lcIvzHx] caught exception while handling client http traffic, closing connection [id: 0x76d5f9f4, L:/127.0.0.1:9200 - R:/127.0.0.1:45696]
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1334) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:926) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:644) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:544) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:498) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:458) [netty-transport-4.1.11.Final.jar:4.1.11.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.11.Final.jar:4.1.11.Final]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_131]
Caused by: javax.net.ssl.SSLException: Received fatal alert: bad_certificate
at sun.security.ssl.Alerts.getSSLException(Alerts.java:208) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1666) ~[?:?]
at sun.security.ssl.SSLEngineImpl.fatal(SSLEngineImpl.java:1634) ~[?:?]
at sun.security.ssl.SSLEngineImpl.recvAlert(SSLEngineImpl.java:1800) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readRecord(SSLEngineImpl.java:1083) ~[?:?]
at sun.security.ssl.SSLEngineImpl.readNetRecord(SSLEngineImpl.java:907) ~[?:?]
at sun.security.ssl.SSLEngineImpl.unwrap(SSLEngineImpl.java:781) ~[?:?]
at javax.net.ssl.SSLEngine.unwrap(SSLEngine.java:624) ~[?:1.8.0_131]
at io.netty.handler.ssl.SslHandler$SslEngineType$3.unwrap(SslHandler.java:254) ~[?:?]
at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1156) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1078) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

I'm seeing this in my log files as well.

6

Let me know if you need more information. I'm not trying to do anything fancy. Just connect to elasticsearch with https

7

OK, Now that you've provided the top half of the error stack trace, it's much clearer.

This is caused by your web browser.

Your browser is not configured to trust the Certificate Authority that your elasticsearch node is using. That is totally normal, certgen is unable to directly generate certificates that browsers trust by default - for that you need to generate a Certificate Signing Request (which certgen can do) and send it off to a public CA.

When the browser encounters a certificate that it doesn't trust, it sends an error response to the server and closes the connection. Elasticsearch reports that error in the logs.

There are a bunch of solutions, but the right solution depends on your environment, and given the nature of these sort of security decisions, I can't really advise you on which is the right choice for your needs. But the options are:

  • Generate a Certificate Signing Request (CSR) and send it to a trusted public CA. Some CAs will charge you for this, some are free, but you need to be able to prove that you are responsible for the domain that you are using for your server hostname.
  • Generate a Certificate Signing Request (CSR) and send it to an internal trusted CA. Many large organisations run their own internal CA for use on their network, and install that CA into the browser/OS trust store as part of their standard desktop environment.
  • Add your elasticsearch (certgen generated) CA to your browser/OS trust store. The instructions vary by OS and browser, but you can find tutorials on the internet. For personal use this is usually OK (though it does have security implications), but if you want others to be able to access your server, then it's probably not a long term viable option.
  • Add a temporary or permanent exception for this certificate in your browser. The instructions vary by browser, and some browsers make it easier than other. You should be able to find instructions on the internet. For personal use this is usually OK, but if you want others to be able to access your server, then it's probably not a long term viable option.
  • Don't worry about browser access. Elasticsearch isn't really intended to be accessed directly from a web browser, so if you're just trying to do this for testing you can use something like curl instead that has simpler options for handling CA trust.
  • Don't run SSL/TLS on the HTTP interface. It's good if you can do it, but as you've found it can also be tricky to get working.
8

Thanks Tim!

I appreciate the time you took to narrow down my issue. Apologies for not providing all the information up front. I wasn't really sure what was important or not.

I remember doing this before on a previous attempt but I guess something else was wrong when I tried.

Thanks again!

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.