#Unable to connect to Elasticsearch

rajushrajan · April 16, 2018, 1:12pm

After enabling xpack in elasticsearch.yml i am geting the below error in kibana.

Bill_McConaghy · April 16, 2018, 1:28pm

Can you connect to that Elasticsearch URL via a CURL command ? I would verify that. Also, look in the Kibana logs and see what errors you find.

rajushrajan · April 16, 2018, 1:38pm

Thank you for you reply.

While connecting using curl -XGET i am geting the below .

and kibana logs shows below error.

Bill_McConaghy · April 16, 2018, 1:44pm

Looks like you have a space before the colon in your kibana.yml file (should be:
xpack.security.enabled: false
so go ahead and address that and restart.

rajushrajan · April 16, 2018, 2:22pm

i have tried that and i am not able to start kibana. I got the same error for last line but could not find any space there

and these are the last lines in my kibana.yml

xpack.monitoring.elasticsearch.url :http://myip:9200/
xpack.monitoring.elasticsearch.username :elastic
xpack.monitoring.elasticsearch.password :
xpack.security.enabled:false
action.auto_create_index:.security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
xpack.security.transport.ssl.enabled:true
xpack.security.transport.ssl.verification_mode:certificate
xpack.security.transport.ssl.keystore.path:config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path:config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled:true
xpack.security.http.ssl.keystore.path:/etc/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path:/etc/elasticsearch/config/certs/elastic-certificates.p12

Bill_McConaghy · April 16, 2018, 2:36pm

In what you have pasted there is a space after other keys before the colon (e.g., 'xpack.monitoring.elasticsearch.url :'), and not after. The syntax specifies the reverse.

rajushrajan · April 16, 2018, 3:00pm

Got it ....Getting some other error....

{"type":"log","@timestamp":"2018-04-16T14:49:13Z","tags":["plugin","warning"],"pid":31057,"path":"/usr/share/kibana/plugins/x-pack","message":"Skipping non-plugin directory at /usr/share/kibana/plugins/x-pack"}
{"type":"log","@timestamp":"2018-04-16T14:49:13Z","tags":["fatal"],"pid":31057,"message":""xpack.monitoring.elasticsearch.url", "xpack.monitoring.elasticsearch.username", "xpack.monitoring.elasticsearch.password", "xpack.security.enabled", "xpack.security.transport.ssl.enabled", "xpack.security.transport.ssl.verification_mode", "xpack.security.transport.ssl.keystore.path", "xpack.security.transport.ssl.truststore.path", "xpack.security.http.ssl.enabled", "xpack.security.http.ssl.keystore.path", "xpack.security.http.ssl.truststore.path", and "action.auto_create_index" settings were not applied. Check for spelling errors and ensure that expected plugins are installed and enabled."}
FATAL CLI ERROR Error: EACCES: permission denied, unlink '/var/run/kibana.pid'
at Error (native)
at fs.unlinkSync (fs.js:1112:18)
at process. (/usr/share/kibana/src/server/pid/index.js:56:26)
at process. (/usr/share/kibana/node_modules/lodash/index.js:7363:25)
at process.g (events.js:292:16)
at emitOne (events.js:101:20)
at process.emit (events.js:188:7)
at process.exit (internal/process.js:166:15)
at Command. (/usr/share/kibana/src/cli/serve/serve.js:70:17)
at throw (native)
at step (/usr/share/kibana/src/cli/serve/serve.js:111:191)
at /usr/share/kibana/src/cli/serve/serve.js:111:402

rajushrajan · April 16, 2018, 3:23pm

i have resolved this error and started kibana. Now i am getting first error again in kibana.

and i have commented the xpack lines in kibana.yml before starting.

Now the log says below

{"type":"log","@timestamp":"2018-04-16T15:25:40Z","tags":["warning","elasticsearch","admin"],"pid":590,"message":"No living connections"}

Bill_McConaghy · April 16, 2018, 3:39pm

I think the errors without the xpack config options commented out are due to xpack not being installed in Kibana? Sounds like it as it is saying those are unrecognized options. The error from Curl above suggests that Elasticsearch is using SSL, so maybe change the elasticsearch url in kibana.yml to SSL?

ikakavas · April 16, 2018, 3:59pm

There are a number of issues with your configuration, I would highly suggest that you go through the documentation and do the configuration step by step, as this greatly simplify the process.
I would start from

That said, You probably have TLS for the http layer enabled in Elasticsearch and/or you are using the wrong passoword. You also have settings from (apparently) logstash and elasticsearch in your kibana.yml for some reason.

These settings are for elasticsearch.yml and should not be in your kibana.yml

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/config/certs/elastic-certificates.p12

These settings are not for kibana.yml either so please remove them :

xpack.monitoring.elasticsearch.url: http://myip:9200/
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password:

What you need is something like:

elasticsearch.url: "https://myip:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "<kibana_user_password_here"

I will once again suggest that you go through the docs and properly start over the configuration so that you get a clearer baseline to work from and if you're still having issues we will be glad to assist further.

rajushrajan · April 19, 2018, 1:34pm

Thank you for the reply.

Any idea why i am getting the below error in elasticsearch log

{"type":"log","@timestamp":"2018-04-19T13:29:19Z","tags":["warning","elasticsearch","admin"],"pid":16473,"message":"No living connections"}

and in kibana still its same error .

ikakavas · April 19, 2018, 2:07pm

Ηi, this is in the kibana.log , not in the elasticsearch.log. It means that Kibana can't talk to Elasticsearch but without any more context it's really hard to know why.

We have all the best intentions to help you with your configuration and get your nodes up and running, but it would be much easier if you provided us with concrete steps on what you are doing and what is your config.

Did you go through the doumentation as suggested ?
Did you fix your configuration ? How does it look like now ?
What are the errors in the logs ( Please share more lines )

rajushrajan · April 19, 2018, 2:14pm

Hi , I am going through the document, i have found that the issue with xpack security configuration.
when i am commenting below lines in elasticsearch.yml kibana is able to connect.

#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.keystore.path: config/certs/elastic-certificates.p12
#xpack.security.http.ssl.truststore.path: config/certs/elastic-certificates.p12

issue is persist when i am uncommenting this lines.

ikakavas · April 19, 2018, 2:17pm

So there is something wrong with your TLS configuration apparently. Can you share the relevant parts of your kibana.yml and elasticsearch.yml ? Also please start elasticsearch with the above 3 lines uncommented and check the elasticsearch.log for indications of an error ( or share the log here so we can help further )

rajushrajan · April 19, 2018, 2:41pm

This is my configuration

kibana.yml

server.host: "myip"
elasticsearch.url: "http://myip:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "password"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-certificates.p12" ]

elasticsearch.yml

cluster.name: elastic-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: myip
discovery.zen.ping.unicast.hosts: ["myip:9300"]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: config/certs/elastic-certificates.p12

#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.keystore.path: config/certs/elastic-certificates.p12
#xpack.security.http.ssl.truststore.path: config/certs/elastic-certificates.p12

Got below error while starting ES after uncommenting:

[2018-04-19T20:06:35,439][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection [id: 0xcddace35, L:0.0.0.0/0.0.0.0:9200 ! R:/ip:32770]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a417574686f72697a6174696f6e3a2042617369632061326c69595735684f6b567359584e3061574e414d54497a0d0a486f73743a2031302e3136392e33332e3139383a393230300d0a436f6e74656e742d4c656e6774683a20300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a417574686f72697a6174696f6e3a2042617369632061326c69595735684f6b567359584e3061574e414d54497a0d0a486f73743a2031302e3136392e33332e3139383a393230300d0a436f6e74656e742d4c656e6774683a20300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

ikakavas · April 19, 2018, 3:07pm

When you uncomment those lines, you enable TLS/SSL for the http layer of Elasticsearch . That means that clients can only communicate to Elasticsearch via http over TLS/SSL . Kibana now has :
elasticsearch.url: "http://myip:9200"
so it will try to connect over plain http, and this is what causes the:

[2018-04-19T20:06:35,439][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection [id: 0xcddace35, L:0.0.0.0/0.0.0.0:9200 ! R:/10.169.33.198:32770]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record:

which clearly states that it received non SSL traffic.

So. you need to change

elasticsearch.url: "http://myip:9200"

to

elasticsearch.url: "https://myip:9200"

when you are uncommenting the lines.

Then, you will have a second problem:

Kibana doesn't currently support PKCS#12 keystores, so the

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-certificates.p12" ]

is invalid. You would need to get the CA certificate in PEM format. Assuming that you have used certutil to get the elastic-certificates.p12, you can get the CA cert in PEM with

openssl pkcs12 -in /etc/kibana/elastic-certificates.p12 -cacerts -nokeys -out /etc/kibana/elastic-ca.pem

and then change

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-certificates.p12" ]

to

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-ca.pem" ]

rajushrajan · April 24, 2018, 8:16am

Hi Thanks for that , the same error has occurred again after the cahnges in the ES log.

[2018-04-24T13:27:28,147][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection [id: 0x5bdb9e41, L:0.0.0.0/0.0.0.0:9200 ! R:/ip:57051]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a2031302e3136392e33332e3139383a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a2031302e3136392e33332e3139383a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

and in kibana showing could not connect

ikakavas · April 24, 2018, 8:30am

Hi. Please read through my post again and perform all the changes suggested

It looks like you didn't do everything, and specifically:

since Kibana still attempts to connect to Elasticsearch over plain http.

rajushrajan · April 24, 2018, 10:03am

I have done the both changes but in kibana it shows like below

ikakavas · April 24, 2018, 10:48am

Screenshots of Kibana are not helping us to assist you as these neither show your configuration, nor the errors you're getting. I'm going to have to make the exact same request I made 5 days ago