Not able to connect to elastic from kibana

Elastic Stack Kibana
1

Hello,
I am trying to set up the OIDC authentication for Kibana in SSO . I have followed the steps from elastic guide.

The CA and HTTP certificates are created as per the details provided in above link.
The elastic is restarted and running post the configuration . However when trying to access through KIbana getting the below error.

**Kibana logs:**

Aug 25 08:32:58 kibana[5916]: {"type":"log","@timestamp":"2023-08-25T06:32:58Z","tags":["warning","elasticsearch","admin"],"pid":5916,"message":"No living connections"}

Aug 25 08:33:01 kibana[5916]: {"type":"log","@timestamp":"2023-08-25T06:33:01Z","tags":["warning","elasticsearch","admin"],"pid":5916,"message":"Unable to revive connection: [https://XX.XX.XXX.XXX:9200](https://xx.xx.xxx.xxx:9200/)"}

Aug 25 08:33:01 kibana[5916]: {"type":"log","@timestamp":"2023-08-25T06:33:01Z","tags":["warning","elasticsearch","admin"],"pid":5916,"message":"No living connections"}

Aug 25 08:33:03 kibana[5916]: {"type":"log","@timestamp":"2023-08-25T06:33:03Z","tags":["warning","elasticsearch","admin"],"pid":5916,"message":"Unable to revive connection: [https://XX.XX.XXX.XXX:9200](https://xx.xx.xxx.xxx:9200/)"}

Aug 25 08:33:03 kibana[5916]: {"type":"log","@timestamp":"2023-08-25T06:33:03Z","tags":["warning","elasticsearch","admin"],"pid":5916,"message":"No living connections"

Here are my yml configurations.

**Elasticsearch.yml**

xpack.security.authc.token.enabled: true

 
#Enable SSL

xpack.security.http.ssl.enabled: true

xpack.security.http.ssl.keystore.path: "http.p12"

#xpack.security.http.ssl.keystore.path: /apps/elastic/elasticsearch/config/http.p12


**Kibana.yml**

elasticsearch.ssl.certificateAuthorities: /apps/elastic/kibana/config/elasticsearch-ca.pem

# The URLs of the Elasticsearch instances to use for all your queries.

elasticsearch.hosts: [[https://XX.XX.XXX.XXX:9200](https://xx.xx.xxx.xxx:9200/)]

Could someone help to confirm why the error is coming.

Regards,
Chitra

2

Hello Team,

Can someone help on it please?

Thanks,
Chitra

3

There not much info to go on (for me). Only thing I can confirm is that Kibana is not able to reach elasticsearch.

You should add a larger excerpt from your logs + increase log level to have more detailed info.

I assume you checked these topics with similar issues?

4

Kibana can reach ES because:

  • ES is not active, check is ES active on port 9200
  • network.host is not correctly set in elasticsearch.yml. Defaults is _local_ -127.0.0.1. You can set to _ site_ or simply 0.0.0.0
  • there is no https on ES or not correctly set. Check can you access curl -k https://<hostorip>:9200
  • firewall in active on ES node(s) - TCP port 9200, you can test by telnet <hostorip> 9200
6

This configuration is in the wrong format.

And this shows up in the error:

Aug 25 08:33:03 kibana[5916]: {"type":"log","@timestamp":"2023-08-25T06:33:03Z","tags":["warning","elasticsearch","admin"],"pid":5916,"message":"Unable to revive connection: [https://XX.XX.XXX.XXX:9200](https://xx.xx.xxx.xxx:9200/)"}

It must be like this:

elasticsearch.hosts: ["https://XX.XX.XXX.XXX:9200"]
7

Hello Leandrojmp,

This is fixed. Here is the latest configuration in kibana.yml.

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["https://XX.XX.XXX.XXX:9200"]

I have updated the error logs post that in the last comment.

Regards,
Chitra

8

Both your log

[2023-08-31T14:37:56,682][WARN ][o.e.h.AbstractHttpServerTransport] [node-1] caught exception while handling client http traffic, closing connection Netty4HttpChannel{localAddress=0.0.0.0/0.0.0.0:9200, remoteAddress=/XX.XX.XXX.XXX:51976}
io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Received fatal alert: unknown_ca

and the curl

*       issuer: CN=Elastic Certificate Tool Autogenerated CA
* NSS error -8172 (SEC_ERROR_UNTRUSTED_ISSUER)
* Peer's certificate issuer has been marked as not trusted by the user.

are clearly stating that you have certificate issues. Your Certificate Authority (CA) is not trusted.

I'll leave it up to leandrojmp & Rios to help you out. I didn't set up certificates myself yet.

9

It still looks like some misconfiguration.

In your elasticsearch.yml you have this:

xpack.security.http.ssl.enabled: true

Is this still configured? If you change any configuration file you need to update it in the post.

With this configuration Elasticsearch will listen with https, but your Kibana logs is showing that Kibana is trying to connect using http.

Aug 31 14:25:04 s01vl9924810 kibana[3250]: {"type":"log","@timestamp":"2023-08-31T12:25:04Z","tags":["warning","elasticsearch","admin"],"pid":3250,"message":"Unable to revive connection: http://55.18.187.149:9200/"}

You need to validate this.

Also, you are using the basic license.

[2023-08-31T14:31:27,059][INFO ][o.e.l.LicenseService ] [node-1] license [56d604fb-d201-4e59-b71e-b70dd4807571] mode [basic] - valid
[2023-08-31T14:31:27,060][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [BASIC]; Security is disabled

SSO does not work with the basic license, you need a paid license or at least set it to use the trial license.

11

Hi Leandrojmp,

I tried enabling the trial version for the elastic using the below commands and response was success.

curl -X POST "http://XX.XX.XXX.XXX:9200/_license/start_trial?acknowledge=true&pretty"

However , after enabling the trail version the elastic is not getting started. This is latest log I have in elastic and I think the latest starts after enabling trial version are not getting recorded in the log itself.

[2023-09-01T09:35:12,137][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [.watches] creating index, cause [auto(bulk api)], templates [.watches], shards [1]/[0], mappings [_doc]
[2023-09-01T09:35:12,427][INFO ][o.e.x.w.WatcherService   ] [node-1] reloading watcher, reason [new local watcher shard allocation ids], cancelled [0] queued tasks
[2023-09-01T09:35:12,453][INFO ][o.e.c.m.MetaDataMappingService] [node-1] [.watches/Tx2Ro_RRSlifc9SCstINAw] update_mapping [_doc]
[2023-09-01T09:36:12,801][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [.triggered_watches] creating index, cause [auto(bulk api)], templates [.triggered_watches], shards [1]/[1], mappings [_doc]
[2023-09-01T09:36:12,803][INFO ][o.e.c.r.a.AllocationService] [node-1] updating number_of_replicas to [0] for indices [.triggered_watches]
[2023-09-01T09:36:13,872][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [.watcher-history-10-2023.09.01] creating index, cause [auto(bulk api)], templates [.watch-history-10], shards [1]/[0], mappings [_doc]
[2023-09-01T09:36:13,978][INFO ][o.e.c.m.MetaDataCreateIndexService] [node-1] [.monitoring-alerts-7] creating index, cause [auto(bulk api)], templates [.monitoring-alerts-7], shards [1]/[0], mappings [_doc]
[2023-09-01T09:36:14,125][INFO ][o.e.x.i.IndexLifecycleTransition] [node-1] moving index [.watcher-history-10-2023.09.01] from [null] to [{"phase":"new","action":"complete","name":"complete"}] in policy [watch-history-ilm-policy]
[2023-09-01T09:36:14,314][INFO ][o.e.c.m.MetaDataMappingService] [node-1] [.watcher-history-10-2023.09.01/UXao6DRqRymgWJ1WoX7Ahg] update_mapping [_doc]
[2023-09-01T09:36:14,539][INFO ][o.e.c.m.MetaDataMappingService] [node-1] [.watcher-history-10-2023.09.01/UXao6DRqRymgWJ1WoX7Ahg] update_mapping [_doc]
[2023-09-01T09:36:14,644][INFO ][o.e.c.m.MetaDataMappingService] [node-1] [.watcher-history-10-2023.09.01/UXao6DRqRymgWJ1WoX7Ahg] update_mapping [_doc]
[2023-09-01T09:37:54,139][INFO ][o.e.n.Node               ] [node-1] stopping ...
[2023-09-01T09:37:54,152][INFO ][o.e.x.w.WatcherService   ] [node-1] stopping watch service, reason [shutdown initiated]
[2023-09-01T09:37:54,153][INFO ][o.e.x.w.WatcherLifeCycleService] [node-1] watcher has stopped and shutdown
[2023-09-01T09:37:54,333][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [node-1] [controller/18317] [Main.cc@150] Ml controller exiting
**[2023-09-01T09:37:54,335][INFO ][o.e.x.m.p.NativeController] [node-1] Native controller process has stopped - no new native processes can be started**
[2023-09-01T09:37:55,010][INFO ][o.e.n.Node               ] [node-1] stopped
[2023-09-01T09:37:55,011][INFO ][o.e.n.Node               ] [node-1] closing ...
[2023-09-01T09:37:55,028][INFO ][o.e.n.Node               ] [node-1] closed

Also this is what I see in the gc.log.0.current.

OpenJDK 64-Bit Server VM (25.382-b05) for linux-amd64 JRE (1.8.0_382-b05), built on Jul 14 2023 06:06:28 by "mockbuild" with gcc 4.8.5 20150623 (Red Hat 4.8.5-44)
Memory: 4k page, physical 32749192k(5670336k free), swap 16781308k(16735520k free)
CommandLine flags: -XX:+AlwaysPreTouch -XX:CMSInitiatingOccupancyFraction=75 -XX:ErrorFile=logs/hs_err_pid%p.log -XX:GCLogFileSize=67108864 -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=data -XX:InitialHeapSize=1073741824 -XX:MaxDirectMemorySize=536870912 -XX:MaxHeapSize=1073741824 -XX:MaxNewSize=357916672 -XX:MaxTenuringThreshold=6 -XX:NewSize=357916672 -XX:NumberOfGCLogFiles=32 -XX:OldPLABSize=16 -XX:OldSize=715825152 -XX:-OmitStackTraceInFastThrow -XX:+PrintGC -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCDateStamps -XX:+PrintGCDetails -XX:+PrintGCTimeStamps -XX:+PrintTenuringDistribution -XX:ThreadStackSize=1024 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseCompressedClassPointers -XX:+UseCompressedOops -XX:+UseConcMarkSweepGC -XX:+UseGCLogFileRotation -XX:+UseParNewGC
2023-09-01T13:45:20.819+0200: 1.310: Total time for which application threads were stopped: 0.0002081 seconds, Stopping threads took: 0.0001060 seconds
2023-09-01T13:45:20.935+0200: 1.425: Total time for which application threads were stopped: 0.0002408 seconds, Stopping threads took: 0.0000989 seconds
2023-09-01T13:45:21.041+0200: 1.531: Total time for which application threads were stopped: 0.0002788 seconds, Stopping threads took: 0.0000841 seconds
Heap
 par new generation   total 314560K, used 89517K [0x00000000c0000000, 0x00000000d5550000, 0x00000000d5550000)
  eden space 279616K,  32% used [0x00000000c0000000, 0x00000000c576b5f8, 0x00000000d1110000)
  from space 34944K,   0% used [0x00000000d1110000, 0x00000000d1110000, 0x00000000d3330000)
  to   space 34944K,   0% used [0x00000000d3330000, 0x00000000d3330000, 0x00000000d5550000)
 concurrent mark-sweep generation total 699072K, used 0K [0x00000000d5550000, 0x0000000100000000, 0x0000000100000000)
 Metaspace       used 12714K, capacity 13034K, committed 13312K, reserved 1060864K
  class space    used 1369K, capacity 1509K, committed 1536K, reserved 1048576K
2023-09-01T13:45:22.041+0200: 2.532: Total time for which application threads were stopped: 0.0002589 seconds, Stopping threads took: 0.0000711 seconds

Regards,
Chitra

12

The information in these logs shows that the node was shutdown, but there is no error in it and the log is incomplete, it does not have everything from the start.

[2023-09-01T09:37:55,028][INFO ][o.e.n.Node ] [node-1] closed

Can you please share your entire elasticsearch.yml? You share only part of it, also share your entire kibana.yml.

Have you started Elasticsearch again? Please run systemctl start elasticsearch and share what happens and what it logs when it starts.

14

Your Elasticsearch service is not running, there is some issue while starting it.

Since this is a systemd error you need to check in the system logs, which should be /var/log/messsages or /var/log/syslog depending on the distribution.

I would suggest that you try to start it again to get fresh logs and then check those files to get the issue on why the service is not starting.

You didn't change the path.logs in the elasticsearch.yml file, so the logs will be stored in the default path, which is /var/log/elasticsearch, you need to check this path for any log file.

16

[2023-09-08T11:34:36,937][INFO ][o.e.x.s.s.SecurityStatusChangeListener] [node-1] Active license is now [TRIAL]; Security is disabled

This log line says that Security is Disabled, did you disable it in elasticsearch.yml ? In version 8 security needs to be explictily disabled.

Your kibana error is still the same, it says that it cannot connect to Elasticsearch, there is nothing else, you need to check the connectivity between the services, this error will happen before any configuration error.

And I Just noticed now that your kibana.yml does not have any information about authenticating in Elasticsearch, Kibana needs to authenticate into Elasticsearch using an internal account or service account.

Which Elastic Guide you followed? Because the security is not configured correctly and your kibana.yml has non-default settings.

I would suggest removing any SSO configuration until your Kibana can connect to your Elasticsearch.

17

Hello ,

I followed the steps from https://www.elastic.co/guide/en/elasticsearch/reference/current/oidc-guide.html.
Here is my Kibana startup log.

-- Logs begin at Wed 2023-09-06 07:42:02 CEST, end at Mon 2023-09-11 07:41:20 CEST. --
Sep 11 07:39:01  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:39:01Z","tags":["warning","elasticsearch","admin"],"pid":22983,"message":"Unable to revive connection: http://XX.XX.XXX.XXX:9200/"}
Sep 11 07:39:13  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:39:13Z","tags":["warning","plugins","licensing"],"pid":22983,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
Sep 11 07:39:43  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:39:43Z","tags":["warning","elasticsearch","data"],"pid":22983,"message":"No living connections"}
Sep 11 07:39:43  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:39:43Z","tags":["warning","plugins","licensing"],"pid":22983,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins-system"],"pid":22983,"message":"Stopping all plugins."}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","bfetch"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","graph"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","apm"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","cloud"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","spaces"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","home"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","share"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","data"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","translations"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","apm_oss"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","security"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","features"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","timelion"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","canvas"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","metrics"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","usageCollection"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","code"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","encryptedSavedObjects"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","infra"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","licensing"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","siem"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","plugins","taskManager"],"pid":22983,"message":"Stopping plugin"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["info","savedobjects-service"],"pid":22983,"message":"Starting saved objects migrations"}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["warning","savedobjects-service"],"pid":22983,"message":"Unable to connect to Elasticsearch. Error: Cluster client cannot be used after it has been closed."}
Sep 11 07:40:40  kibana[22983]: {"type":"log","@timestamp":"2023-09-11T05:40:40Z","tags":["fatal","root"],"pid":22983,"message":"Error: Cluster client cannot be used after it has been closed.\n    at ClusterClient.assertIsNotClosed (/apps/elastic/kibana/src/core/server/elasticsearch/cluster_client.js:187:13)\n    at ClusterClient._defineProperty [as callAsInternalUser] (/apps/elastic/kibana/src/core/server/elasticsearch/cluster_client.js:108:12)\n    at callAsInternalUser (/apps/elastic/kibana/src/core/server/elasticsearch/elasticsearch_service.js:72:29)"}
Sep 11 07:40:40  kibana[22983]: FATAL  Error: Cluster client cannot be used after it has been closed.
Sep 11 07:40:47  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:47Z","tags":["info","plugins-service"],"pid":30394,"message":"Plugin \"case\" is disabled."}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","config","deprecation"],"pid":30394,"message":"Environment variable CONFIG_PATH is deprecated. It has been replaced with KIBANA_PATH_CONF pointing to a config folder"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins-system"],"pid":30394,"message":"Setting up [37] plugins: [taskManager,siem,licensing,infra,encryptedSavedObjects,code,usageCollection,metrics,canvas,timelion,features,security,apm_oss,translations,reporting,uiActions,data,navigation,status_page,share,newsfeed,kibana_legacy,management,dev_tools,inspector,expressions,visualizations,embeddable,advancedUiActions,dashboard_embeddable_container,home,spaces,cloud,apm,graph,eui_utils,bfetch]"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","taskManager"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","siem"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","licensing"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","infra"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","encryptedSavedObjects"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","plugins","encryptedSavedObjects","config"],"pid":30394,"message":"Generating a random key for xpack.encryptedSavedObjects.encryptionKey. To be able to decrypt encrypted saved objects attributes after restart, please set xpack.encryptedSavedObjects.encryptionKey in kibana.yml"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","code"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","usageCollection"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","metrics"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","canvas"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","timelion"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","features"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","security"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","plugins","security","config"],"pid":30394,"message":"Generating a random key for xpack.security.encryptionKey. To prevent sessions from being invalidated on restart, please set xpack.security.encryptionKey in kibana.yml"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","plugins","security","config"],"pid":30394,"message":"Session cookies will be transmitted over insecure connections. This is not recommended."}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","apm_oss"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","translations"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","data"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","share"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","home"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","spaces"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","cloud"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","apm"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","graph"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","plugins","bfetch"],"pid":30394,"message":"Setting up plugin"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["info","savedobjects-service"],"pid":30394,"message":"Waiting until all Elasticsearch nodes are compatible with Kibana before starting saved objects migrations..."}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["error","elasticsearch","data"],"pid":30394,"message":"Request error, retrying\nHEAD http://XX.XX.XXX.XXX:9200/.apm-agent-configuration => socket hang up"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["error","elasticsearch","data"],"pid":30394,"message":"Request error, retrying\nGET http://XX.XX.XXX.XXX:9200/_xpack => socket hang up"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["error","elasticsearch","admin"],"pid":30394,"message":"Request error, retrying\nGET http://XX.XX.XXX.XXX:9200/_nodes?filter_path=nodes.*.version%2Cnodes.*.http.publish_address%2Cnodes.*.ip => socket hang up"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","elasticsearch","data"],"pid":30394,"message":"Unable to revive connection: http://XX.XX.XXX.XXX:9200/"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","elasticsearch","data"],"pid":30394,"message":"No living connections"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","plugins","licensing"],"pid":30394,"message":"License information could not be obtained from Elasticsearch due to Error: No Living connections error"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","elasticsearch","admin"],"pid":30394,"message":"Unable to revive connection: http://XX.XX.XXX.XXX:9200/"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","elasticsearch","admin"],"pid":30394,"message":"No living connections"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["error","savedobjects-service"],"pid":30394,"message":"Unable to retrieve version information from Elasticsearch nodes."}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","elasticsearch","data"],"pid":30394,"message":"Unable to revive connection: http://XX.XX.XXX.XXX:9200/"}
Sep 11 07:40:53  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:53Z","tags":["warning","elasticsearch","data"],"pid":30394,"message":"No living connections"}
Sep 11 07:40:53  kibana[30394]: Could not create APM Agent configuration: No Living connections
Sep 11 07:40:55  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:55Z","tags":["warning","elasticsearch","admin"],"pid":30394,"message":"Unable to revive connection: http://XX.XX.XXX.XXX:9200/"}
Sep 11 07:40:55  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:55Z","tags":["warning","elasticsearch","admin"],"pid":30394,"message":"No living connections"}
Sep 11 07:40:58  kibana[30394]: {"type":"log","@timestamp":"2023-09-11T05:40:58Z","tags":["warning","elasticsearch","admin"],"pid":30394,"message":"Unable to revive connection: http://XX.XX.XXX.XXX:9200/"}

Earlier without the SSO configuration I was able to connect to Elastic from Kibana without any issues. However after enabling the trial version am getting this error.
Please find below the config files without SSO below.

elasticsearch.yml

# ======================== Elasticsearch Configuration =========================
#
# NOTE: Elasticsearch comes with reasonable defaults for most settings.
#       Before you set out to tweak and tune the configuration, make sure you
#       understand what are you trying to accomplish and the consequences.
#
# The primary way of configuring a node is via this file. This template lists
# the most important settings you may want to configure for a production cluster.
#
# Please consult the documentation for further information on configuration options:
# https://www.elastic.co/guide/en/elasticsearch/reference/index.html
#
# ---------------------------------- Cluster -----------------------------------
#
# Use a descriptive name for your cluster:
#
#cluster.name: my-application
#
# ------------------------------------ Node ------------------------------------
#
# Use a descriptive name for the node:
#
node.name: node-1
#
# Add custom attributes to the node:
#
node.attr.rack: r1
#node.data: false
#
# ----------------------------------- Paths ------------------------------------
#
# Path to directory where to store the data (separate multiple locations by comma):
#
#path.data: /path/to/data
#
# Path to log files:
#
#path.logs: /path/to/logs
#
#Path to Snapshot:
#
path.repo: /apps/elastic/elasticsearch 
#
# ----------------------------------- Memory -----------------------------------
#
# Lock the memory on startup:
#
bootstrap.memory_lock: true
#
# Make sure that the heap size is set to about half the memory available
# on the system and that the owner of the process is allowed to use this
# limit.
#
# Elasticsearch performs poorly when the system is swapping the memory.
#
# ---------------------------------- Network -----------------------------------
#
# Set the bind address to a specific IP (IPv4 or IPv6):
#
network.host: XX.XX.XXX.XXX
#
# Set a custom port for HTTP:
#
http.port: 9200
#
# For more information, consult the network module documentation.
#
# --------------------------------- Discovery ----------------------------------
#
# Pass an initial list of hosts to perform discovery when this node is started:
# The default list of hosts is ["127.0.0.1", "[::1]"]
#
discovery.seed_hosts: ["XX.XX.XXX.XXX"]
#
# Bootstrap the cluster using an initial set of master-eligible nodes:
#
cluster.initial_master_nodes: ["node-1"]

#
# For more information, consult the discovery and cluster formation module documentation.
#
# ---------------------------------- Gateway -----------------------------------
#
# Block initial recovery after a full cluster restart until N nodes are started:
#
#gateway.recover_after_nodes: 3
#
# For more information, consult the gateway module documentation.
#
# ---------------------------------- Various -----------------------------------
#
# Require explicit names when deleting indices:
#
#action.destructive_requires_name: true

indices.lifecycle.poll_interval: 60s

kibana.yml

# Kibana is served by a back end server. This setting specifies the port to use.
server.port: 5601

# Specifies the address to which the Kibana server will bind. IP addresses and host names are both valid values.
# The default is 'localhost', which usually means remote machines will not be able to connect.
# To allow connections from remote users, set this parameter to a non-loopback address.
server.host: abcd.in.net.intra 

# Enables you to specify a path to mount Kibana at if you are running behind a proxy.
# Use the `server.rewriteBasePath` setting to tell Kibana if it should remove the basePath
# from requests it receives, and to prevent a deprecation warning at startup.
# This setting cannot end in a slash.
#server.basePath: ""

# Specifies whether Kibana should rewrite requests that are prefixed with
# `server.basePath` or require that they are rewritten by your reverse proxy.
# This setting was effectively always `false` before Kibana 6.3 and will
# default to `true` starting in Kibana 7.0.
#server.rewriteBasePath: false

# The maximum payload size in bytes for incoming server requests.
#server.maxPayloadBytes: 1048576

# The Kibana server's name.  This is used for display purposes.
server.name: "Amanda-Muse"

# The URLs of the Elasticsearch instances to use for all your queries.
elasticsearch.hosts: ["http://XX.XX.XXX.XXX:9200"]

# When this setting's value is true Kibana uses the hostname specified in the server.host
# setting. When the value of this setting is false, Kibana uses the hostname of the host
# that connects to this Kibana instance.
elasticsearch.preserveHost: true

# Kibana uses an index in Elasticsearch to store saved searches, visualizations and
# dashboards. Kibana creates a new index if the index doesn't already exist.
kibana.index: ".kibana"

# The default application to load.
#kibana.defaultAppId: "home"

# If your Elasticsearch is protected with basic authentication, these settings provide
# the username and password that the Kibana server uses to perform maintenance on the Kibana
# index at startup. Your Kibana users still need to authenticate with Elasticsearch, which
# is proxied through the Kibana server.

#elasticsearch.username: "elastic"
#elasticsearch.username: "kibana"
#elasticsearch.password: "pass"

# Enables SSL and paths to the PEM-format SSL certificate and SSL key files, respectively.
# These settings enable SSL for outgoing requests from the Kibana server to the browser.
#server.ssl.enabled: false
#server.ssl.certificate: /path/to/your/server.crt
#server.ssl.key: /path/to/your/server.key

# Optional settings that provide the paths to the PEM-format SSL certificate and key files.
# These files are used to verify the identity of Kibana to Elasticsearch and are required when
# xpack.security.http.ssl.client_authentication in Elasticsearch is set to required.
#elasticsearch.ssl.certificate: /path/to/your/client.crt
#elasticsearch.ssl.key: /path/to/your/client.key

# Optional setting that enables you to specify a path to the PEM file for the certificate
# authority for your Elasticsearch instance.
#elasticsearch.ssl.certificateAuthorities: /apps/elastic/kibana/config/elasticsearch-ca.pem
#elasticsearch.ssl.certificateAuthorities: "elasticsearch-ca.pem"

# To disregard the validity of SSL certificates, change this setting's value to 'none'.
#elasticsearch.ssl.verificationMode: full

# Time in milliseconds to wait for Elasticsearch to respond to pings. Defaults to the value of
# the elasticsearch.requestTimeout setting.
#elasticsearch.pingTimeout: 1500

# Time in milliseconds to wait for responses from the back end or Elasticsearch. This value
# must be a positive integer.
#elasticsearch.requestTimeout: 30000


# List of Kibana client-side headers to send to Elasticsearch. To send *no* client-side
# headers, set this value to [] (an empty list).
#elasticsearch.requestHeadersWhitelist: [ authorization ]

# Header names and values that are sent to Elasticsearch. Any custom headers cannot be overwritten
# by client-side headers, regardless of the elasticsearch.requestHeadersWhitelist configuration.
#elasticsearch.customHeaders: {}

# Time in milliseconds for Elasticsearch to wait for responses from shards. Set to 0 to disable.
#elasticsearch.shardTimeout: 30000

# Time in milliseconds to wait for Elasticsearch at Kibana startup before retrying.
#elasticsearch.startupTimeout: 5000

# Logs queries sent to Elasticsearch. Requires logging.verbose set to true.
#elasticsearch.logQueries: false

# Specifies the path where Kibana creates the process ID file.
#pid.file: /var/run/kibana.pid

# Enables you specify a file where Kibana stores log output.
#logging.dest: stdout

# Set the value of this setting to true to suppress all logging output.
#logging.silent: false

# Set the value of this setting to true to suppress all logging output other than error messages.
#logging.quiet: false

# Set the value of this setting to true to log all events, including system usage information
# and all requests.
#logging.verbose: false

# Set the interval in milliseconds to sample system and process performance
# metrics. Minimum is 100ms. Defaults to 5000.
#ops.interval: 5000

# Specifies locale to be used for all localizable strings, dates and number formats.
# Supported languages are the following: English - en , by default , Chinese - zh-CN .
#i18n.locale: "en"

Both elastic and kibana are using version 7.6.2 . Is it because of the trial version in elastic ??

Regards,
Chitra

19

Hello ,

I was able to connect to elastic to Kibana (without SSO config) successfully by having the security disabled.

# Enable security features
xpack.security.enabled: false

I followed the steps from the link for SSO OIDC setup https://www.elastic.co/guide/en/elasticsearch/reference/current/oidc-guide.html.
Both elastic and kibana are using version 7.6.2 .

Regards,
Chitra

20

Hello Leandrojmp,
Could you help to understand what went wrong in the SSO OIDC config please ?

As I mentioned in my previous post, without SSO config my KIBANA connectivity is working as expected. However with SSO OIDC it is not working. I followed the steps from the link for SSO OIDC setup https://www.elastic.co/guide/en/elasticsearch/reference/current/oidc-guide.html .
Both elastic and kibana are using version 7.6.2 .

Also, after your last comment I understood that Kibana.yml was missing OIDC set up. So I fixed it by adding below .

# The default application to load.
#kibana.defaultAppId: "home"
#Enable OIDC
xpack.security.authc.providers:
oidc.oidc1:
   order: 0
   realm: "oidc1"

xpack.security.session.idleTimeout: "30m"
xpack.security.session.lifespan: "7d"

But still KIBANA is not able to connect to elastic .

Regards,
Chitra

21

Unfortunatelly I have no experience configuring SSO in Elasticsearch/Kibana.

But you may be missing some things.

First, 7.6.2 is pretty old, it already reached EOL date and you should not use it, you should use a more recent version, which are 7.17 or 8.10.

Second, since you are not using the current version, you should follow the documentation for your specific version, you can change the version in the left side of the documentation page, but in this case it will be this one.

Third, did you set basic security frist? You should first configure basic security, this is needed for Kibana to be able to connect to Elasticsearch when security is enabled, after you have basic security configured and are able to authenticate in Kibana you can try to configure SSO.

The last elasticsearch.yml you shared does not have basic security configured, it also does not have security enable, you need to enable security and configure basic security first, this is required.

The documentation about configuring security on a 7.6 cluster is this one.

You basically need to:

  • Enable security in your cluster
  • Create the passwords for the internal users
  • Configure Kibana to connect to Elasticsearch with security enabled

You need this to be working before trying to configure SSO.

22

Hello Leandrojmp,

I tried basic security authentication with User and password in trial version. It is working fine. I am able to connect to elastic from KIBANA as well.
However for OIDC set up , am still facing issues. It looks like CERTIFICATE TRUST issue. Not sure if it is because of self signed certificate.

Could you confirm if we can use KIBANA in SSO with LDAP mode of authentication ?
If that is supported , i can try that as it doesn't need any CERTIFICATES.

Regards,
Chitra

23

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.