Kibana throws errors after SSl configuration in configmap kibana.yml

Abu84 · March 23, 2020, 9:57am

Continuing the discussion from Kibana logging errors after SSl configuration:

here is my configuration. This is my complete kibana.yml configuration.

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
server.ssl.enabled: true
server.ssl.certificate: /cert/test.crt
server.ssl.key: /cert/test.key

markov00 · March 23, 2020, 1:29pm

Hi @Abu84
Are you on the same situation mentioned on the other thread (http ES and https Kibana)?
could you please post the Kibana/ES versions and the logs from both?

Abu84 · March 23, 2020, 2:24pm

yes..its TLS enabled only in Kibana..and HTTP on Elastic
Below are the versions..

Component	Helm Chart	App Version
Elastic Search	bitnami/elasticsearch	7.5.1
Kibana	bitnami/kibana	7.5.1

Logs below of Kibana:

{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["error","elasticsearch","admin"],"pid":36,"message":"Request error, retrying\nGET http://elasticsearch-coordinating-only:9200/.kibana => connect ECONNREFUSED 10.233.15.115:9200"}
{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["warning","elasticsearch","admin"],"pid":36,"message":"Unable to revive connection: http://elasticsearch-coordinating-only:9200/"}
{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["warning","elasticsearch","admin"],"pid":36,"message":"No living connections"}
{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["warning","migrations"],"pid":36,"message":"Unable to connect to Elasticsearch. Error: No Living connections"}

markov00 · March 23, 2020, 2:36pm

Are you sure the following host and port is reachable from your Kibana machine: http://elasticsearch-coordinating-only:9200

try with:
curl -X GET "elasticsearch-coordinating-only:9200/_cat/health?v&pretty"

Abu84 · March 23, 2020, 3:17pm

Elastic endpoint is not exposed as NodePort, its cluster IP. Hence its not reachable outside cluster.

Also when ssl in kibana is disabled the connection to elastic is successful and Kibana GUI is accessible and able to display elastic indexes and data.

Only when SSL enabled in kibana, these errors are seen and kibana pod keeps crashing

ELASTIC logs


14:32:39.39 INFO  ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-03-23T14:32:41,530][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-coordinating-only-598449456b-5jtdc] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/mapper/vg00-root)]], net usable_space [84.2gb], net total_space [99.4gb], types [xfs]
[2020-03-23T14:32:41,531][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-coordinating-only-598449456b-5jtdc] heap size [123.7mb], compressed ordinary object pointers [true]
[2020-03-23T14:32:41,534][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] node name [elasticsearch-coordinating-only-598449456b-5jtdc], node ID [aiwbokgnSQS8Xj96f4Ty3A], cluster name [elastic]
[2020-03-23T14:32:41,539][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] version[7.5.1], pid[1], build[oss/tar/3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96/2019-12-16T22:57:37.835892Z], OS[Linux/3.10.0-1062.4.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/11.0.5/11.0.5+10]
[2020-03-23T14:32:41,539][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] JVM home [/opt/bitnami/java]
[2020-03-23T14:32:41,539][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-18344636417337590589, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-03-23T14:32:42,212][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [aggs-matrix-stats]
[2020-03-23T14:32:42,212][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [analysis-common]
[2020-03-23T14:32:42,212][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [ingest-common]
............................
[2020-03-23T14:32:42,214][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [rank-eval]
[2020-03-23T14:32:42,214][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [reindex]
[2020-03-23T14:32:42,215][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [repository-url]
[2020-03-23T14:32:42,215][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [transport-netty4]
[2020-03-23T14:32:42,215][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] no plugins loaded
[2020-03-23T14:32:44,866][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-coordinating-only-598449456b-5jtdc] using discovery type [zen] and seed hosts providers [settings]
[2020-03-23T14:32:45,259][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] initialized
[2020-03-23T14:32:45,259][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] starting ...
[2020-03-23T14:32:45,393][INFO ][o.e.t.TransportService   ] [elasticsearch-coordinating-only-598449456b-5jtdc] publish_address {10.233.102.157:9300}, bound_addresses {0.0.0.0:9300}
[2020-03-23T14:32:45,415][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-coordinating-only-598449456b-5jtdc] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-03-23T14:32:45,549][WARN ][o.e.t.TransportService   ] [elasticsearch-coordinating-only-598449456b-5jtdc] Transport response handler not found of id [2]
[2020-03-23T14:32:47,754][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-coordinating-only-598449456b-5jtdc] master node changed {previous [], current [{elasticsearch-master-1}{twTVHm5cSie7LYSFqu1lVg}{1WoIPpIQSyG0ZhCO--2tdw}{10.233.102.108}{10.233.102.108:9300}{m}]}, added {{elasticsearch-data-0}{KYWrbDa3RMCTeV7dd-82Jg}{1a3oXIGlTH-pFbv60-1HFw}{10.233.102.181}{10.233.102.181:9300}{d},{elasticsearch-coordinating-only-598449456b-4r9vf}{fe4rIGLTRKeeZqGJ5bl6yw}{NG4xEdQqSJip8ypHFRzDtQ}{10.233.117.169}{10.233.117.169:9300},{elasticsearch-data-1}{BE3yVYXEQySVUqKzcLc6QA}{bt3t5DFYReKRoXfT0_SjuA}{10.233.117.158}{10.233.117.158:9300}{d},{elasticsearch-master-0}{AvD6SK3yQpqFfATSNVtFlQ}{2-DypbMjQKWJNQvoD_UUpQ}{10.233.117.166}{10.233.117.166:9300}{m},{elasticsearch-master-1}{twTVHm5cSie7LYSFqu1lVg}{1WoIPpIQSyG0ZhCO--2tdw}{10.233.102.108}{10.233.102.108:9300}{m}}, term: 42, version: 637, reason: ApplyCommitRequest{term=42, version=637, sourceNode={elasticsearch-master-1}{twTVHm5cSie7LYSFqu1lVg}{1WoIPpIQSyG0ZhCO--2tdw}{10.233.102.108}{10.233.102.108:9300}{m}}
[2020-03-23T14:32:47,793][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-coordinating-only-598449456b-5jtdc] publish_address {10.233.102.157:9200}, bound_addresses {0.0.0.0:9200}
[2020-03-23T14:32:47,794][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] started```

LizaD · March 23, 2020, 5:25pm

Hi @Abu84,

Welcome to our community! So I tried standalone packages for Elasticsearch and Kibana for 7.5.1 and I was able to connect Kibana TLS using your config to Elasticsearch basic. I generated the certs from Elasticsearch. So I am thinking this is not a Kibana issue, but mostly like a setup issue. Can you give me more information on the setup, looks like you are using bitnami helm, is that correct? I can try to reproduce your setup and/or maybe also try posting in the ECK forum.

Thanks!
Liza

Abu84 · March 24, 2020, 5:43am

@ markov00 and @LizaD..
Thank you for your replies.

We have generated the certs using openssl..Is that fine or you need to generate certs using elastic search util only?

If so, could you please provide steps on how we generate it and export so that it can be used for kibana..

SETUP

Yes , we are using bitnami helm..Its kubernetes 1.15 version..and helm 2.12 version..

Abu84 · March 24, 2020, 6:19am

typo corrected..

elasticsearch.hosts: ["**http://elasticsearch-coordinating-only:9200**"]

Abu84 · March 24, 2020, 9:40am

Looking for a solution..yet..plz do suggest..

LizaD · March 24, 2020, 5:30pm

Hi @Abu84,

I am not familiar with bitnami helm setup so not sure, I have more familiarity with our ECK, if you are interested in learning more here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-kibana.html.

For generating the certs in Elasticsearch you can run bin/elasticsearch-certgen for 7.5.1 and it will generate the certs.

So I was able to get elasticsearch running under bitnami helm but I am still trying to figure out the certs part (I have not done it before). Will keep you posted.

Thanks!
Liza

LizaD · March 24, 2020, 6:05pm

Hi @Abu84:

I wanted to share a bit more information that I have learned a long the way, so we also maintain our own helm charts and that is what we recommend, you can find information in below. We will be able to provide more support for you with that one

For the bitnami helm setup we also found this file:

github.com

bitnami/charts/blob/master/bitnami/elasticsearch/values-production.yaml

## Global Docker image parameters
## Please, note that this will override the image parameters, including dependencies, configured to use the global value
## Current available global Docker image parameters: imageRegistry and imagePullSecrets
##
global:
  # imageRegistry: myRegistryName
  # imagePullSecrets:
  #   - myRegistryKeySecretName
  # storageClass: myStorageClass
  ## Coordinating name to be used in the Kibana subchart (service name)
  ##
  coordinating:
    name: coordinating-only
  kibanaEnabled: true

## Bitnami Elasticsearch image version
## ref: https://hub.docker.com/r/bitnami/elasticsearch/tags/
##
image:
  registry: docker.io

This file has been truncated. show original

It shows a place to mount the certs and what to place in the config. Again, still have not tried it yet, but hopefully the information helps a bit.

Thanks,
Liza

Abu84 · March 26, 2020, 6:37am

HI,

This is what I found on more research, that there is an issue with security integration in bitnami helm.

Also With regards to tools like elasticsearch-certgen and elasticsearch-certutil, these utilities are not available in bitnami.

Abu84 · March 26, 2020, 6:42am

HI,

We have tried out the options you provided for bitnami helm, unfortunately we don't see it working.

As per your suggestions, to try elastic helm from elastic repo rather than bitnami, can you please clarify the below mentioned on helm hub page for elastic such that we can take a decision to move to elastic repo helm.

***This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.***

NOTE: bitnami helm chart has no such limitations mentioned, thats why we had opted for it.

Michelle_Sausa · March 26, 2020, 4:47pm

@Abu84 Thanks for your inquiry. We have considered the charts to be technically mature for some time now. They have remained in beta to provided ample time and opportunity for user feedback and any proposed use cases/usage scenarios which might require breaking changes to the charts to surface. We are now actively discussing possible timing for moving the charts to GA status.

Abu84 · March 27, 2020, 8:53am

Thank you for the clarification. Could you please let us know an approximate date, such that we can take a decision and move to these GA level charts from elastic repo.

Vernon · April 9, 2020, 6:04am

The Kibana server reads properties from the kibana.yml file on startup. The location of this file differs depending on how you installed Kibana. For example, if you installed Kibana from an archive distribution ( .tar.gz or .zip ), by default it is in $KIBANA_HOME/config .

Vernon · April 10, 2020, 8:57am

Any one have other option?

system · May 8, 2020, 8:57am

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Logstash SSL/TLS error Logstash	1	343	October 3, 2023
{"statusCode":404,"error":"Not Found","message":"Not Found"} after security Kibana	14	16291	November 23, 2020
Unable to retrieve version information from Elasticsearch nodes. security_exception Kibana elastic-stack-security	5	19495	December 16, 2023
Elastic Search Configuration Problem with Kibana.yml Kibana	4	24	October 11, 2024
How to config kibana if xpack.security.http.ssl.enabled is false Kibana elastic-stack-security , docker	8	4294	July 19, 2022

Kibana throws errors after SSl configuration in configmap kibana.yml

Related Topics