Encrypt Communication b/w kibana and elasticsearch

I have 8 node of elasticsearch cluster 3 master 3 hot node and 2 warm node.All these node is running on https://ip:9200
i have set following parameter in elasticsearch.yml

xpack.security.enabled: true
#xpack.security.audit.enabled: true
xpack.security.transport.ssl.enabled: true
#xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.client_authentication: optional
xpack.security.transport.ssl.key: certs/em1/em1.key
xpack.security.transport.ssl.certificate: certs/em1/em1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  certs/em1/em1.key
xpack.security.http.ssl.certificate: certs/em1/em1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

My kibana is showing server is not ready yet

elasticsearch.hosts: ["https://em1:9200","https://em2:9200"]
elasticsearch.username: "user_name"
elasticsearch.password: "password"
elasticsearch.ssl.verificationMode: certificate
elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca/ca.crt" ]

I'd ran this command bin/elasticsearch-certutil cert --days 7300 --keysize 2048 --keep-ca-key --pem --in ./instances.yaml --out ./instances.zip
in instance.yaml file i have provided list of ips and node name of all elasticsearch node

elasticsearch.ssl.certificateAuthorities should have .pem certificate but i have only ca.crt and ca.key.How can i generate the pem file ? i had also ran this command openssl x509 -in ca/ca.key -inform d -outform PEM -out elastic.pem but it can't read the crt file.
I have copied ca.crt file from elasticsearch node to kibana.But the kibana is not running

In kibana log file

{"type":"log","@timestamp":"2021-02-25T22:53:38+05:30","tags":["warning","plugins","licensing"],"pid":16037,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2021-02-25T22:53:08+05:30","tags":["warning","elasticsearch","monitoring"],"pid":16037,"message":"Unable to revive connection: https://em2:9200/"}
{"type":"log","@timestamp":"2021-02-25T22:53:08+05:30","tags":["warning","elasticsearch","monitoring"],"pid":16037,"message":"Unable to revive connection: https://em1:9200/"}

Looks like your elasticsearch cluster did not start successfully. What do the elasticsearch logs say ?

.crt and .pem are just file suffixes ,there is no difference. You can use ca.crt and ca.pem interchangeably, you can rename the file with no issues. Tim wrote an explanation recently, feel free to take a look in Elastic stack issues Certificates and Kibana is not ready yet - #3 by TimV

yes you are right i didn't notice that
elasticsearch log

[2021-02-26T11:49:29,245][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [em1] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/node1:9200, remoteAddress=/client:53192}

when i do https://node1:9200 in broswer provide the username and password and elasticsearch page opens and when i do this with other node ```https://node2:9200" enter the username and password... it don't accept username and password although the credential is right but it is not accepting.

hey @ikakavas i made some changes in elasticsearch.yml


xpack.security.enabled: true
#xpack.security.audit.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.client_authentication: required
xpack.security.transport.ssl.key: certs/em1/em1.key
xpack.security.transport.ssl.certificate: certs/em1/em1.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt

xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  certs/em1/em1.key
xpack.security.http.ssl.certificate: certs/em1/em1.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

in elasticsearch.log file

received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/ip_of_master_node:9200, remoteAddress=/kibana_ip:35824}

Here is kibana.yml file

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: "ip"

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# Specifies the public URL at which Kibana is available for end users. If
# `server.basePath` is configured this URL should end with the same basePath.
#server.publicBaseUrl: ""

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
#server.name: "your-hostname"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://master_node_ip:9200"]

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
#kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.
#elasticsearch.username: "kibana_system"
elasticsearch.username: "elastic"
elasticsearch.password: "Test@123"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
elasticsearch.ssl.certificate: /etc/kibana/certs/ca/ca.crt
elasticsearch.ssl.key: /etc/kibana/certs/ca/ca.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca/ca.crt" ]

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000

# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /run/kibana/kibana.pid

# Enables you to specify a file where Kibana stores log output.
logging.dest: /var/log/kibana/kibana.log
logging.rotate:
  enabled: true
  everyBytes: 10485760

#
# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

kibana logs

{"type":"log","@timestamp":"2021-02-27T14:08:34+05:30","tags":["error","elasticsearch","monitoring"],"pid":1591,"message":"Request error, retrying\nGET https://master_node_ip:9200/_xpack?accept_enterprise=true => unable to verify the first certificate"}
{"type":"log","@timestamp":"2021-02-27T14:08:34+05:30","tags":["warning","elasticsearch","monitoring"],"pid":1591,"message":"Unable to revive connection: https://master_node_ip:9200/"}
{"type":"log","@timestamp":"2021-02-27T14:08:34+05:30","tags":["warning","elasticsearch","monitoring"],"pid":1591,"message":"No living connections"}
{"type":"log","@timestamp":"2021-02-27T14:08:34+05:30","tags":["warning","plugins","licensing"],"pid":1591,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
{"type":"log","@timestamp":"2021-02-27T14:08:34+05:30","tags":["warning","plugins","monitoring","monitoring"],"pid":1591,"message":"X-Pack Monitoring Cluster Alerts will not be available: No Living connections"}
But still kibana is not running

The issue is solved i made the following changes

elasticsearch.ssl.certificateAuthorities: [ "/etc/kibana/certs/ca/ca.crt" ]
elasticsearch.ssl.verificationMode: full

For what is worth and for future readers of this post, the fact that the error you were getting was

received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/ip_of_master_node:9200, remoteAddress=/kibana_ip:35824}`

it most probably means that you had

elasticsearch.hosts: ["http://master_node_ip:9200"]

in your kibana.yml and you changed it to

elasticsearch.hosts: ["https://master_node_ip:9200"]

and this is what resolved your issue.

when generating certificates for all the nodes i am using instances.yaml file in which i was using ip addresses and when using certificate in elasticsearch.yml ,kibana.yml or logstash.yml it didn't work..in elasticsearch log file it was saying em2(hostname_of_master_node) does not match in your certificate then i change em2 to ip in kibana.yml ,logstash.yml and metricbeat.yml

[2021-03-01T19:01:21,878][WARN ][o.e.x.s.t.n.SecurityNetty4HttpServerTransport] [eh3] received plaintext http traffic on an https channel, closing connection Netty4HttpChannel{localAddress=/hot_node3_ip:9200, remoteAddress=/hot_node2_ip:56846

i did the telnet hot_node3_ip 9200 from hot_node_2 server

when generating certificates for all the nodes i am using instances.yaml file in which i was using ip addresses and when using certificate in elasticsearch.yml ,kibana.yml or logstash.yml it didn't work..in elasticsearch log file it was saying em2(hostname_of_master_node) does not match in your certificate then i change em2 to ip in kibana.yml ,logstash.yml and metricbeat.yml

Apologies, I don;t understand what you are asking, can you be more explicit? What are you trying to do and what happens instead?

[eh3] received plaintext http traffic on an https channel

i did the telnet hot_node3_ip 9200 from hot_node_2 server

Telnet(the binary) can't handle HTTP over TLS. Elasticsearch sees something connecting to it on its port that doesn't do TLS and throws an error, this is expected. You need to use another client to interact with Elasticsearch on the HTTP layer, I suggest you use curl.

I regenerate the certificate for hot_node_3 but kibana and metricbeat cannot connect to this node or metricbeat cannot access events
kibana.log


{"type":"log","@timestamp":"2021-03-01T17:13:09+05:30","tags":["error","elasticsearch","monitoring"],"pid":11225,"message":"Request error, retrying\nGET https://1hot_node3_ip:9200/_xpack?accept_enterprise=true => connect EHOSTUNREACH hot_node3_ip:9200"}
{"type":"log","@timestamp":"2021-03-01T17:13:10+05:30","tags":["warning","elasticsearch","monitoring"],"pid":11225,"message":"Unable to revive connection: https://hot_node3_ip:9200/"}
{"type":"log","@timestamp":"2021-03-01T17:13:10+05:30","tags":["warning","elasticsearch","monitoring"],"pid":11225,"message":"No living connections"}
{"type":"log","@timestamp":"2021-03-01T17:13:10+05:30","tags":["warning","plugins","licensing"],"pid":11225,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}

but i can curl

[root@eh3 elasticsearch]# curl -XGET --cacert /etc/elasticsearch/certs/ca/ca.crt -u aniket "https://hot_node3_ip:9200"
Enter host password for user 'aniket':
{
  "name" : "eh3",
  "cluster_name" : "hot-warm",
  "cluster_uuid" : "sUe_7CNMSvmyhjg0FRWnJA",
  "version" : {
    "number" : "7.11.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "ff17057114c2199c9c1bbecc727003a907c0db7a",
    "build_date" : "2021-02-15T13:44:09.394032Z",
    "build_snapshot" : false,
    "lucene_version" : "8.7.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

elasticsearch.yml file (hot_node)


xpack.security.enabled: true
#xpack.security.audit.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: full
xpack.security.transport.ssl.key: certs/eh3/eh3.key
xpack.security.transport.ssl.certificate: certs/eh3/eh3.crt
xpack.security.transport.ssl.certificate_authorities: certs/ca/ca.crt
xpack.security.transport.ssl.client_authentication: required
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.key:  certs/eh3/eh3.key
xpack.security.http.ssl.certificate: certs/eh3/eh3.crt
xpack.security.http.ssl.certificate_authorities: certs/ca/ca.crt

Are you running curl from the same machine where kibana is running ? If not, could it be some network issue or firewall that prohibits the host where kibana runs to communicate with the host where elasticseararch runs?

yes it was just because of network issue.. i restarted the server and i am not seeing any warning

while generating certficate for all nodes of elasticsearch using bin/elasticsearch-certutil cert --days 7300 --keysize 2048 --keep-ca-key --pem --in ./instances.yaml --out ./instances.zip
instances.yml


instances:
  - name: 'em1'
    ip: [ 'xx.xx.xx.xx' ]
  - name: 'em2'
    ip: [ 'xx.xx.xx.xx' ]
  - name: 'em3'
    ip: [ 'xx.xx.xx.xx' ]
  - name: 'eh1'
    ip: [ 'xx.xx.xx.xx' ]
  - name: 'eh2'
    ip: [ 'xx.xx.xx.xx' ]
  - name: 'eh3'
    ip: [ 'xx.xx.xx.xx' ]
  - name: 'ew1'
    ip: [ 'xx.xx.xx.xx' ]

The output of this file is ca.crt ca.key and all nodes crt and key file
In kibana.yml when i provide elasticsearch.hosts: ["https://em1:9200",https://em2:9200"]
then kibana will not work when i write elasticsearch.hosts: ["https://em1_ip:9200",https://em2_ip:9200"] then it will work
same with metricbeat

Please don't merge questions and answers between different posts. It might help you in the end, but it complicates things for people that try to help you and provides no value for anyone else that might have the same problems in the future and will happen to read your posts and our replies.

This is usually never enough. What "will not work" means? As I asked above, state what happens, what is the error and what you were expecting to happen instead. This will help you get better answers from other people, because they won't have to guess.

In this case, your problem is that you generate certificates that only have the IP address as SAN. So when a client ( i.e kibana ) tries to connect to elasticsearch at https://em1:9200 it can't verify that the certificate is valid for em1, it only knows it is valid for em1_ip.

You need to change your instances.yml to

instances:
  - name: 'em1'
    ip: [ 'xx.xx.xx.xx' ]
    dns: ['em1']
  - name: 'em2'
    ip: [ 'xx.xx.xx.xx' ]
    dns: ['em2']
  - name: 'em3'
    ip: [ 'xx.xx.xx.xx' ]
    dns: ['em3']
  - name: 'eh1'
    ip: [ 'xx.xx.xx.xx' ]
    dns: ['eh1']
  - name: 'eh2'
    ip: [ 'xx.xx.xx.xx' ]
    dns: ['eh2']
  - name: 'eh3'
    ip: [ 'xx.xx.xx.xx' ]
    dns: ['eh3']
  - name: 'ew1'
    dns: ['ew1']
    ip: [ 'xx.xx.xx.xx' ]
1 Like

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.