#Unable to connect to Elasticsearch

After enabling xpack in elasticsearch.yml i am geting the below error in kibana.

Can you connect to that Elasticsearch URL via a CURL command ? I would verify that. Also, look in the Kibana logs and see what errors you find.

Thank you for you reply.

While connecting using curl -XGET i am geting the below .

image

and kibana logs shows below error.

Looks like you have a space before the colon in your kibana.yml file (should be:
xpack.security.enabled: false
so go ahead and address that and restart.

i have tried that and i am not able to start kibana. I got the same error for last line but could not find any space there

and these are the last lines in my kibana.yml

xpack.monitoring.elasticsearch.url :http://myip:9200/
xpack.monitoring.elasticsearch.username :elastic
xpack.monitoring.elasticsearch.password :
xpack.security.enabled:false
action.auto_create_index:.security,.monitoring*,.watches,.triggered_watches,.watcher-history*,.ml*
xpack.security.transport.ssl.enabled:true
xpack.security.transport.ssl.verification_mode:certificate
xpack.security.transport.ssl.keystore.path:config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path:config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled:true
xpack.security.http.ssl.keystore.path:/etc/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path:/etc/elasticsearch/config/certs/elastic-certificates.p12

In what you have pasted there is a space after other keys before the colon (e.g., 'xpack.monitoring.elasticsearch.url :'), and not after. The syntax specifies the reverse.

Got it ....Getting some other error....

{"type":"log","@timestamp":"2018-04-16T14:49:13Z","tags":["plugin","warning"],"pid":31057,"path":"/usr/share/kibana/plugins/x-pack","message":"Skipping non-plugin directory at /usr/share/kibana/plugins/x-pack"}
{"type":"log","@timestamp":"2018-04-16T14:49:13Z","tags":["fatal"],"pid":31057,"message":""xpack.monitoring.elasticsearch.url", "xpack.monitoring.elasticsearch.username", "xpack.monitoring.elasticsearch.password", "xpack.security.enabled", "xpack.security.transport.ssl.enabled", "xpack.security.transport.ssl.verification_mode", "xpack.security.transport.ssl.keystore.path", "xpack.security.transport.ssl.truststore.path", "xpack.security.http.ssl.enabled", "xpack.security.http.ssl.keystore.path", "xpack.security.http.ssl.truststore.path", and "action.auto_create_index" settings were not applied. Check for spelling errors and ensure that expected plugins are installed and enabled."}
FATAL CLI ERROR Error: EACCES: permission denied, unlink '/var/run/kibana.pid'
at Error (native)
at fs.unlinkSync (fs.js:1112:18)
at process. (/usr/share/kibana/src/server/pid/index.js:56:26)
at process. (/usr/share/kibana/node_modules/lodash/index.js:7363:25)
at process.g (events.js:292:16)
at emitOne (events.js:101:20)
at process.emit (events.js:188:7)
at process.exit (internal/process.js:166:15)
at Command. (/usr/share/kibana/src/cli/serve/serve.js:70:17)
at throw (native)
at step (/usr/share/kibana/src/cli/serve/serve.js:111:191)
at /usr/share/kibana/src/cli/serve/serve.js:111:402

i have resolved this error and started kibana. Now i am getting first error again in kibana.

and i have commented the xpack lines in kibana.yml before starting.

Now the log says below

{"type":"log","@timestamp":"2018-04-16T15:25:40Z","tags":["warning","elasticsearch","admin"],"pid":590,"message":"No living connections"}

I think the errors without the xpack config options commented out are due to xpack not being installed in Kibana? Sounds like it as it is saying those are unrecognized options. The error from Curl above suggests that Elasticsearch is using SSL, so maybe change the elasticsearch url in kibana.yml to SSL?

There are a number of issues with your configuration, I would highly suggest that you go through the documentation and do the configuration step by step, as this greatly simplify the process.
I would start from

  1. Install X-Pack in Elasticsearch
  2. Install X-Pack in Kibana
  3. Setup TLS for your cluster

That said, You probably have TLS for the http layer enabled in Elasticsearch and/or you are using the wrong passoword. You also have settings from (apparently) logstash and elasticsearch in your kibana.yml for some reason.

These settings are for elasticsearch.yml and should not be in your kibana.yml

xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: config/certs/elastic-certificates.p12
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.keystore.path: /etc/elasticsearch/config/certs/elastic-certificates.p12
xpack.security.http.ssl.truststore.path: /etc/elasticsearch/config/certs/elastic-certificates.p12

These settings are not for kibana.yml either so please remove them :

xpack.monitoring.elasticsearch.url: http://myip:9200/
xpack.monitoring.elasticsearch.username: elastic
xpack.monitoring.elasticsearch.password: 

What you need is something like:

elasticsearch.url: "https://myip:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "<kibana_user_password_here"

I will once again suggest that you go through the docs and properly start over the configuration so that you get a clearer baseline to work from and if you're still having issues we will be glad to assist further.

Thank you for the reply.

Any idea why i am getting the below error in elasticsearch log

{"type":"log","@timestamp":"2018-04-19T13:29:19Z","tags":["warning","elasticsearch","admin"],"pid":16473,"message":"No living connections"}

and in kibana still its same error .

Ηi, this is in the kibana.log , not in the Elasticsearch.log. It means that Kibana can't talk to Elasticsearch but without any more context it's really hard to know why.

We have all the best intentions to help you with your configuration and get your nodes up and running, but it would be much easier if you provided us with concrete steps on what you are doing and what is your config.

  • Did you go through the doumentation as suggested ?
  • Did you fix your configuration ? How does it look like now ?
  • What are the errors in the logs ( Please share more lines )

Hi , I am going through the document, i have found that the issue with xpack security configuration.
when i am commenting below lines in elasticsearch.yml kibana is able to connect.

#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.keystore.path: config/certs/elastic-certificates.p12
#xpack.security.http.ssl.truststore.path: config/certs/elastic-certificates.p12

issue is persist when i am uncommenting this lines.

So there is something wrong with your TLS configuration apparently. Can you share the relevant parts of your kibana.yml and elasticsearch.yml ? Also please start elasticsearch with the above 3 lines uncommented and check the elasticsearch.log for indications of an error ( or share the log here so we can help further )

This is my configuration

kibana.yml

server.host: "myip"
elasticsearch.url: "http://myip:9200"
elasticsearch.username: "kibana"
elasticsearch.password: "password"
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-certificates.p12" ]

elasticsearch.yml

cluster.name: elastic-cluster
node.name: node-1
path.data: /var/lib/elasticsearch
path.logs: /var/log/elasticsearch
network.host: myip
discovery.zen.ping.unicast.hosts: ["myip:9300"]
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: config/certs/elastic-certificates.p12
xpack.security.transport.ssl.truststore.path: config/certs/elastic-certificates.p12

#xpack.security.http.ssl.enabled: true
#xpack.security.http.ssl.keystore.path: config/certs/elastic-certificates.p12
#xpack.security.http.ssl.truststore.path: config/certs/elastic-certificates.p12

Got below error while starting ES after uncommenting:

[2018-04-19T20:06:35,439][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection [id: 0xcddace35, L:0.0.0.0/0.0.0.0:9200 ! R:/ip:32770]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a417574686f72697a6174696f6e3a2042617369632061326c69595735684f6b567359584e3061574e414d54497a0d0a486f73743a2031302e3136392e33332e3139383a393230300d0a436f6e74656e742d4c656e6774683a20300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 48454144202f20485454502f312e310d0a417574686f72697a6174696f6e3a2042617369632061326c69595735684f6b567359584e3061574e414d54497a0d0a486f73743a2031302e3136392e33332e3139383a393230300d0a436f6e74656e742d4c656e6774683a20300d0a436f6e6e656374696f6e3a206b6565702d616c6976650d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

When you uncomment those lines, you enable TLS/SSL for the http layer of Elasticsearch . That means that clients can only communicate to Elasticsearch via http over TLS/SSL . Kibana now has :
elasticsearch.url: "http://myip:9200"
so it will try to connect over plain http, and this is what causes the:

[2018-04-19T20:06:35,439][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection [id: 0xcddace35, L:0.0.0.0/0.0.0.0:9200 ! R:/10.169.33.198:32770]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 

which clearly states that it received non SSL traffic.

So. you need to change

elasticsearch.url: "http://myip:9200" 

to

elasticsearch.url: "https://myip:9200" 

when you are uncommenting the lines.

Then, you will have a second problem:

Kibana doesn't currently support PKCS#12 keystores, so the

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-certificates.p12" ]

is invalid. You would need to get the CA certificate in PEM format. Assuming that you have used certutil to get the elastic-certificates.p12, you can get the CA cert in PEM with

openssl pkcs12 -in /etc/kibana/elastic-certificates.p12 -cacerts -nokeys -out /etc/kibana/elastic-ca.pem

and then change

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-certificates.p12" ]

to

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/elastic-ca.pem" ]
1 Like

Hi Thanks for that , the same error has occurred again after the cahnges in the ES log.

[2018-04-24T13:27:28,147][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection [id: 0x5bdb9e41, L:0.0.0.0/0.0.0.0:9200 ! R:/ip:57051]
io.netty.handler.codec.DecoderException: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a2031302e3136392e33332e3139383a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:459) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:265) ~[netty-codec-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:340) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1359) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:362) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:348) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:935) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:134) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:645) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:545) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:499) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:459) [netty-transport-4.1.16.Final.jar:4.1.16.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:858) [netty-common-4.1.16.Final.jar:4.1.16.Final]
at java.lang.Thread.run(Thread.java:745) [?:1.8.0_91]
Caused by: io.netty.handler.ssl.NotSslRecordException: not an SSL/TLS record: 474554202f20485454502f312e310d0a486f73743a2031302e3136392e33332e3139383a393230300d0a557365722d4167656e743a20476f2d687474702d636c69656e742f312e310d0a4163636570743a206170706c69636174696f6e2f6a736f6e0d0a4163636570742d456e636f64696e673a20677a69700d0a0d0a
at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1106) ~[?:?]
at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1162) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:489) ~[?:?]
at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:428) ~[?:?]
... 15 more

and in kibana showing could not connect

Hi. Please read through my post again and perform all the changes suggested

It looks like you didn't do everything, and specifically:

since Kibana still attempts to connect to Elasticsearch over plain http.

I have done the both changes but in kibana it shows like below

Screenshots of Kibana are not helping us to assist you as these neither show your configuration, nor the errors you're getting. I'm going to have to make the exact same request I made 5 days ago