Kibana throws errors after SSl configuration in configmap kibana.yml

Elastic Stack Kibana
1

Continuing the discussion from Kibana logging errors after SSl configuration:

here is my configuration. This is my complete kibana.yml configuration.

server.host: "0.0.0.0"
elasticsearch.hosts: ["http://localhost:9200"]
server.ssl.enabled: true
server.ssl.certificate: /cert/test.crt
server.ssl.key: /cert/test.key
2

Hi @Abu84
Are you on the same situation mentioned on the other thread (http ES and https Kibana)?
could you please post the Kibana/ES versions and the logs from both?

3

yes..its TLS enabled only in Kibana..and HTTP on Elastic
Below are the versions..

Component Helm Chart App Version
Elastic Search bitnami/elasticsearch 7.5.1
Kibana bitnami/kibana 7.5.1

Logs below of Kibana:

{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["error","elasticsearch","admin"],"pid":36,"message":"Request error, retrying\nGET http://elasticsearch-coordinating-only:9200/.kibana => connect ECONNREFUSED 10.233.15.115:9200"}
{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["warning","elasticsearch","admin"],"pid":36,"message":"Unable to revive connection: http://elasticsearch-coordinating-only:9200/"}
{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["warning","elasticsearch","admin"],"pid":36,"message":"No living connections"}
{"type":"log","@timestamp":"2020-03-23T09:16:22Z","tags":["warning","migrations"],"pid":36,"message":"Unable to connect to Elasticsearch. Error: No Living connections"}

4

Are you sure the following host and port is reachable from your Kibana machine: http://elasticsearch-coordinating-only:9200

try with:
curl -X GET "elasticsearch-coordinating-only:9200/_cat/health?v&pretty"

5

Elastic endpoint is not exposed as NodePort, its cluster IP. Hence its not reachable outside cluster.

Also when ssl in kibana is disabled the connection to elastic is successful and Kibana GUI is accessible and able to display elastic indexes and data.

Only when SSL enabled in kibana, these errors are seen and kibana pod keeps crashing

ELASTIC logs


14:32:39.39 INFO  ==> ** Starting Elasticsearch **
OpenJDK 64-Bit Server VM warning: Option UseConcMarkSweepGC was deprecated in version 9.0 and will likely be removed in a future release.
[2020-03-23T14:32:41,530][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-coordinating-only-598449456b-5jtdc] using [1] data paths, mounts [[/bitnami/elasticsearch/data (/dev/mapper/vg00-root)]], net usable_space [84.2gb], net total_space [99.4gb], types [xfs]
[2020-03-23T14:32:41,531][INFO ][o.e.e.NodeEnvironment    ] [elasticsearch-coordinating-only-598449456b-5jtdc] heap size [123.7mb], compressed ordinary object pointers [true]
[2020-03-23T14:32:41,534][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] node name [elasticsearch-coordinating-only-598449456b-5jtdc], node ID [aiwbokgnSQS8Xj96f4Ty3A], cluster name [elastic]
[2020-03-23T14:32:41,539][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] version[7.5.1], pid[1], build[oss/tar/3ae9ac9a93c95bd0cdc054951cf95d88e1e18d96/2019-12-16T22:57:37.835892Z], OS[Linux/3.10.0-1062.4.1.el7.x86_64/amd64], JVM[AdoptOpenJDK/OpenJDK 64-Bit Server VM/11.0.5/11.0.5+10]
[2020-03-23T14:32:41,539][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] JVM home [/opt/bitnami/java]
[2020-03-23T14:32:41,539][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] JVM arguments [-Des.networkaddress.cache.ttl=60, -Des.networkaddress.cache.negative.ttl=10, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -XX:-OmitStackTraceInFastThrow, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dio.netty.allocator.numDirectArenas=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Djava.locale.providers=COMPAT, -Xms128m, -Xmx128m, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -Djava.io.tmpdir=/tmp/elasticsearch-18344636417337590589, -XX:+HeapDumpOnOutOfMemoryError, -XX:HeapDumpPath=data, -XX:ErrorFile=logs/hs_err_pid%p.log, -Xlog:gc*,gc+age=trace,safepoint:file=logs/gc.log:utctime,pid,tags:filecount=32,filesize=64m, -XX:MaxDirectMemorySize=67108864, -Des.path.home=/opt/bitnami/elasticsearch, -Des.path.conf=/opt/bitnami/elasticsearch/config, -Des.distribution.flavor=oss, -Des.distribution.type=tar, -Des.bundled_jdk=true]
[2020-03-23T14:32:42,212][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [aggs-matrix-stats]
[2020-03-23T14:32:42,212][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [analysis-common]
[2020-03-23T14:32:42,212][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [ingest-common]
............................
[2020-03-23T14:32:42,214][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [rank-eval]
[2020-03-23T14:32:42,214][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [reindex]
[2020-03-23T14:32:42,215][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [repository-url]
[2020-03-23T14:32:42,215][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] loaded module [transport-netty4]
[2020-03-23T14:32:42,215][INFO ][o.e.p.PluginsService     ] [elasticsearch-coordinating-only-598449456b-5jtdc] no plugins loaded
[2020-03-23T14:32:44,866][INFO ][o.e.d.DiscoveryModule    ] [elasticsearch-coordinating-only-598449456b-5jtdc] using discovery type [zen] and seed hosts providers [settings]
[2020-03-23T14:32:45,259][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] initialized
[2020-03-23T14:32:45,259][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] starting ...
[2020-03-23T14:32:45,393][INFO ][o.e.t.TransportService   ] [elasticsearch-coordinating-only-598449456b-5jtdc] publish_address {10.233.102.157:9300}, bound_addresses {0.0.0.0:9300}
[2020-03-23T14:32:45,415][INFO ][o.e.b.BootstrapChecks    ] [elasticsearch-coordinating-only-598449456b-5jtdc] bound or publishing to a non-loopback address, enforcing bootstrap checks
[2020-03-23T14:32:45,549][WARN ][o.e.t.TransportService   ] [elasticsearch-coordinating-only-598449456b-5jtdc] Transport response handler not found of id [2]
[2020-03-23T14:32:47,754][INFO ][o.e.c.s.ClusterApplierService] [elasticsearch-coordinating-only-598449456b-5jtdc] master node changed {previous [], current [{elasticsearch-master-1}{twTVHm5cSie7LYSFqu1lVg}{1WoIPpIQSyG0ZhCO--2tdw}{10.233.102.108}{10.233.102.108:9300}{m}]}, added {{elasticsearch-data-0}{KYWrbDa3RMCTeV7dd-82Jg}{1a3oXIGlTH-pFbv60-1HFw}{10.233.102.181}{10.233.102.181:9300}{d},{elasticsearch-coordinating-only-598449456b-4r9vf}{fe4rIGLTRKeeZqGJ5bl6yw}{NG4xEdQqSJip8ypHFRzDtQ}{10.233.117.169}{10.233.117.169:9300},{elasticsearch-data-1}{BE3yVYXEQySVUqKzcLc6QA}{bt3t5DFYReKRoXfT0_SjuA}{10.233.117.158}{10.233.117.158:9300}{d},{elasticsearch-master-0}{AvD6SK3yQpqFfATSNVtFlQ}{2-DypbMjQKWJNQvoD_UUpQ}{10.233.117.166}{10.233.117.166:9300}{m},{elasticsearch-master-1}{twTVHm5cSie7LYSFqu1lVg}{1WoIPpIQSyG0ZhCO--2tdw}{10.233.102.108}{10.233.102.108:9300}{m}}, term: 42, version: 637, reason: ApplyCommitRequest{term=42, version=637, sourceNode={elasticsearch-master-1}{twTVHm5cSie7LYSFqu1lVg}{1WoIPpIQSyG0ZhCO--2tdw}{10.233.102.108}{10.233.102.108:9300}{m}}
[2020-03-23T14:32:47,793][INFO ][o.e.h.AbstractHttpServerTransport] [elasticsearch-coordinating-only-598449456b-5jtdc] publish_address {10.233.102.157:9200}, bound_addresses {0.0.0.0:9200}
[2020-03-23T14:32:47,794][INFO ][o.e.n.Node               ] [elasticsearch-coordinating-only-598449456b-5jtdc] started```
6

Hi @Abu84,

Welcome to our community! So I tried standalone packages for Elasticsearch and Kibana for 7.5.1 and I was able to connect Kibana TLS using your config to Elasticsearch basic. I generated the certs from Elasticsearch. So I am thinking this is not a Kibana issue, but mostly like a setup issue. Can you give me more information on the setup, looks like you are using bitnami helm, is that correct? I can try to reproduce your setup and/or maybe also try posting in the ECK forum.

Thanks!
Liza

7

@ markov00 and @LizaD..
Thank you for your replies.

We have generated the certs using openssl..Is that fine or you need to generate certs using elastic search util only?

If so, could you please provide steps on how we generate it and export so that it can be used for kibana..

SETUP

Yes , we are using bitnami helm..Its kubernetes 1.15 version..and helm 2.12 version..

8

typo corrected..

elasticsearch.hosts: ["**http://elasticsearch-coordinating-only:9200**"]

9

Looking for a solution..yet..plz do suggest..

10

Hi @Abu84,

I am not familiar with bitnami helm setup so not sure, I have more familiarity with our ECK, if you are interested in learning more here: https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-kibana.html.

For generating the certs in Elasticsearch you can run bin/elasticsearch-certgen for 7.5.1 and it will generate the certs.

So I was able to get elasticsearch running under bitnami helm but I am still trying to figure out the certs part (I have not done it before). Will keep you posted.

Thanks!
Liza

11

Hi @Abu84:

I wanted to share a bit more information that I have learned a long the way, so we also maintain our own helm charts and that is what we recommend, you can find information in below. We will be able to provide more support for you with that one :slight_smile:

For the bitnami helm setup we also found this file:

It shows a place to mount the certs and what to place in the config. Again, still have not tried it yet, but hopefully the information helps a bit.

Thanks,
Liza

12

HI,

This is what I found on more research, that there is an issue with security integration in bitnami helm.

Also With regards to tools like elasticsearch-certgen and elasticsearch-certutil, these utilities are not available in bitnami.

13

HI,

We have tried out the options you provided for bitnami helm, unfortunately we don't see it working.

As per your suggestions, to try elastic helm from elastic repo rather than bitnami, can you please clarify the below mentioned on helm hub page for elastic such that we can take a decision to move to elastic repo helm.

***This functionality is in beta and is subject to change. The design and code is less mature than official GA features and is being provided as-is with no warranties. Beta features are not subject to the support SLA of official GA features.***

NOTE: bitnami helm chart has no such limitations mentioned, thats why we had opted for it.

15

@Abu84 Thanks for your inquiry. We have considered the charts to be technically mature for some time now. They have remained in beta to provided ample time and opportunity for user feedback and any proposed use cases/usage scenarios which might require breaking changes to the charts to surface. We are now actively discussing possible timing for moving the charts to GA status.

2 Likes
16

Thank you for the clarification. Could you please let us know an approximate date, such that we can take a decision and move to these GA level charts from elastic repo.

17

The Kibana server reads properties from the kibana.yml file on startup. The location of this file differs depending on how you installed Kibana. For example, if you installed Kibana from an archive distribution ( .tar.gz or .zip ), by default it is in $KIBANA_HOME/config .

18

Any one have other option?

19

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.