Cannot start enterprise search - pre-flight check to Kibana timed out

[Sheereen (Sheereen Hamza)]

Hi,

I am using Elasticsearch 8.8.1 with Kibana (dev mode) 8.8.1.
I installed EnterpriseSearch 8.8.1 (Enterprise Search 8.8.1 | Elastic) and installed the same.
Made the following configurations in /usr/share/enterprise-search/config/enterprise-search.yml.
(both es and enterprise installed on same machine (ES-IP), kibana on different machine(KIBANA-IP))

ent_search.external_url: http://<ES-IP>:3002
ent_search.listen_host: <ES-IP>
ent_search.listen_port: 3002

allow_es_settings_modification: true
secret_management.encryption_keys: ['1716ebb87f98c290a673460573f002dd2b37aa8a5f7970d4af7187140b81c0a6']
elasticsearch.username: elastic
elasticsearch.password: <password>
elasticsearch.host: https://<ES-IP>:9200

elasticsearch.ssl.enabled: true
elasticsearch.ssl.certificate_authority: /usr/share/enterprise-search/config/certs/http_ca.crt

kibana.host: http://<KIBANA-IP>:5600
kibana.external_url: http://<KIBANA-IP>:5600 

I get the following error when I start EnterpriseSearch using bin/enterpriseSearch

Could not log error: uninitialized constant Swiftype::ExceptionTracking at /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:115:in `run!'
        bin/enterprise-search-internal:15:in `<main>'
Unexpected exception while running Enterprise Search:
Error: Connect to <KIBANA-IP>:5600 [/<KIBANA-IP>] failed: connect timed out at

Full log

[root@vm04-es-cluster enterprise-search]# bin/enterprise-search
Found java executable in PATH
Java version detected: 11.0.19 (major version: 11)
Enterprise Search is starting...
ERROR: ld.so: object 'libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[2023-08-08T11:10:24.404+00:00][2011477][4004][app-server][INFO]: Elastic Enterprise Search version=8.8.1, JRuby version=9.3.3.0, Ruby version=2.6.8, Rails version=6.0.6.1
[2023-08-08T11:10:28.750+00:00][2011477][4004][app-server][INFO]: Performing pre-flight checks for Elasticsearch running on https://<ES-IP>:9200...
WARNING: An illegal reflective access operation has occurred
WARNING: Illegal reflective access by org.jruby.javasupport.binding.ConstantField (file:/usr/share/enterprise-search/lib/war/lib/jruby-core-9.3.3.0-complete.jar) to field sun.security.x509.X509CertImpl.SIG
WARNING: Please consider reporting this to the maintainers of org.jruby.javasupport.binding.ConstantField
WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
WARNING: All illegal access operations will be denied in a future release
[2023-08-08T11:10:29.495+00:00][2011477][4004][app-server][INFO]: Elasticsearch cluster is ready
[2023-08-08T11:10:29.497+00:00][2011477][4004][app-server][INFO]: Successfully connected to Elasticsearch
[2023-08-08T11:10:29.562+00:00][2011477][4004][app-server][INFO]: Successfully loaded Elasticsearch plugin information for all nodes
[2023-08-08T11:10:29.574+00:00][2011477][4004][app-server][INFO]: Elasticsearch running with an active basic license
[2023-08-08T11:10:29.608+00:00][2011477][4004][app-server][INFO]: Elasticsearch API key service is enabled
[2023-08-08T11:10:29.620+00:00][2011477][4004][app-server][INFO]: Elasticsearch will be used for authentication
[2023-08-08T11:10:29.621+00:00][2011477][4004][app-server][INFO]: Elasticsearch looks healthy and configured correctly to run Enterprise Search
[2023-08-08T11:10:29.624+00:00][2011477][4004][app-server][INFO]: Performing pre-flight checks for Kibana running on http://<KIBANA-IP>:5600...
Could not log error: uninitialized constant Swiftype::ExceptionTracking at /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:115:in `run!'
        bin/enterprise-search-internal:15:in `<main>'
Unexpected exception while running Enterprise Search:
Error: Connect to <KIBANA-IP>:5600 [/<KIBANA-IP>] failed: connect timed out at

Is it necessary for Kibana and EnterpriseSearch to be installed on the same Linux machine?

Thanks

[Sean_Story]

Hi @Sheereen ,

Sorry you're having an issue getting Enterprise Search up and running.

I noticed that your installation path was /usr/share/enterprise-search/, would I be right to guess that you're running with Docker?

It looks to me like the root issue is

Error: Connect to <KIBANA-IP>:5600 [/<KIBANA-IP>] failed

This kind of thing is a common speedbump when running with Docker, though it could be an issue with any setup. My guess is that there are some networking barriers between the host running Enterprise Search and the host running Kibana. You can test this theory by SSHing into the host/container where Enterprise Search is running, and attempting to do a

curl -v -k -u elastic:<password> <KIBANA-IP>:5600

Note, it's possible to have set up allowing traffic from your Kibana host to Enterprise Search, but not vice-versa. This error is stating that Enterprise Search can't make requests to Kibana.

[Sheereen (Sheereen Hamza)]

Thanks for the reply.

I have installed using RPM. And yes, the problem is due to connectivity issues...I am not able to ping the Kibana host from the host where EnterpriseSearch is installed.

If u look at the log (same I had shared before), there is an error at the beginning of it

[root@vm04-es-cluster enterprise-search]# bin/enterprise-search
Found java executable in PATH
Java version detected: 11.0.19 (major version: 11)
Enterprise Search is starting...
ERROR: ld.so: object 'libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.
[2023-08-08T11:10:24.404+00:00][2011477][4004][app-server][INFO]: Elastic Enterprise Search version=8.8.1, JRuby version=9.3.3.0, Ruby version=2.6.8, Rails version=6.0.6.1
...

ERROR: ld.so: object 'libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

Why do I get this error? I had initially installed EnterpriseSearch 8.9 and 8.0 unknowingly, then I did not have this error. Now I have 8.8.1 since ES and Kibana is 8.8.1, and then I get this error.

Because of this error, I am not able to start EnterpriseSearch as a service using systemctl.. I have to always go to /usr/share/enterprise-search and then do bin/enterprise-search and keep the terminal up an running.

I get that error when I do bin/enterprise-search as well, as I had shared before... but then after sometime it continues to rest of the process. But with systemctl, it gets stuck there and restarts.

Thanks

[Sean_Story]

If u look at the log (same I had shared before), there is an error at the beginning of it
ERROR: ld.so: object 'libjemalloc.so' from LD_PRELOAD cannot be preloaded (cannot open shared object file): ignored.

This looks like an environment-specific error. Googling it, sounds like you might have some C-related environment variables that aren't set quite right. However, I don't think this has anything to do with Enterprise Search not being able to establish a network connection with Kibana.

Because of this error, I am not able to start EnterpriseSearch as a service using systemctl. . I have to always go to /usr/share/enterprise-search and then do bin/enterprise-search

I am skeptical of this, and think there may be a misunderstanding. That error is just the last of the STDOUT/STDERR messages that Enterprise Search produces on startup. After those first few moments, all output gets routed through our logger, which, in your RPM setup, likely does not go to STDOUT but to another file. Your systemctl status enteprise-search.service output shows that it's still active (running), so I don't think that ld.so error crashed the server when you ran from RPM, you're just not looking at the relevant logs.

I am not able to ping the Kibana host from the host where EnterpriseSearch is installed.

Let's resolve this issue first. Once your'e able to hit the Kibana host from the Enterprise Search host with Curl, lets see if that fixes running Enterprise Search with bin/enterprise-search. Once that's working, we can see if there are any lingering issues when you run using systemctl.