Enterprise-search not starting new install of 8.6.2

Elastic Search
1

Elastic search is running no problem
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:password https://127.0.0.1:9200
Response looks good.
But when starting Enterprise Search get error see below. Elastic search and Enterprise Search are on the same stand alone system. While these are warnings and infos it is evedent that enterprise search is making a request to Elasticsearch which is throwing an exeception. The request gets "Failed to connect to Elasticsearch backend"

==========================================================================
enterprise-search.yml
==========================================================================
## ================= Elastic Enterprise Search Configuration ==================
#
secret_management.encryption_keys: [3A16224D695239597DFC19268304E0B58CCA9210C7B65B5DA1C21B10F8CAE9E0]
# ---------------------------------- TLS/SSL ----------------------------------
# Configure TLS/SSL encryption.
#
ent_search.listen_host: 0.0.0.0
ent_search.ssl.enabled: true
ent_search.ssl.keystore.path: "/usr/share/enterprise-search/config/rofccs901a.mayo.edu.jks"
ent_search.ssl.keystore.password: 'Meyers92'
ent_search.ssl.keystore.key_password: 'Meyers92'
# ------------------------------- Hosting & Network ---------------------------
#
ent_search.external_url: https://rofccs901a.mayo.edu:3002
#
# ------------------------------- Elasticsearch -------------------------------
#
#ent_search.auth.default.source: standard
elasticsearch.host: https://127.0.0.1:9200
elasticsearch.username: elastic
elasticsearch.password: 1234567
#
# ---------------------------------- Logging ----------------------------------
#
log_directory: /app/log/enterprise-search
#
log_level:debug
#
log_format: default
====================================================================
ERROR in enterprise log
====================================================================
[2023-03-23T20:20:22.579+00:00][55850][4004][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
[2023-03-23T20:20:22.580+00:00][55850][4004][app-server][INFO]: [pre-flight] Error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:179: 127.0.0.1:9200 failed to respond (Faraday::ClientError)

[2023-03-23T20:20:27.594+00:00][55850][4004][es][DEBUG]: {
  "request": {
    "url": "http://127.0.0.1:9200/",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.10.2"
    },
    "params": null,
    "body": null
  },
  "exception": "/usr/share/enterprise-search/lib/war/lib/middleware/request_logging_middleware.class:55: 127.0.0.1:9200 failed to respond (Faraday::ClientError)\n",
  "duration": 6.5,
  "stack": [
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:651:in `block in get_raw'",
    "/usr/share/enterprise-search/lib/war/lib/apm_helpers.class:41:in `es_action_instrument'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:749:in `instrument'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:650:in `get_raw'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:134:in `check_es_connection_with_retries!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:32:in `check_es_connection!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:19:in `block in run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:18:in `run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:14:in `run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:291:in `configure_elasticsearch!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:270:in `configure!'",
    "/usr/share/enterprise-search/lib/war/config/application.class:20:in `<main>'",
    "/usr/share/enterprise-search/lib/war/config/application.rb:1:in `<main>'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:36:in `initialize'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:148:in `run_supported_command'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:130:in `run_command'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2023-03-23T20:20:27.596+00:00][55850][4004][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
2

is Elasticsearch running on https? From your configuration, looks like you're configured to use http://127.0.0.1:9200 for elasticsearch.

3

Yep you are correct, sorry. I have updated the post to https and restarted. Getting same error but now request URL is https://127.0.0.1:9200. I have been working on this for some time now and have tried many things all have failed. I am assuming alot of people are successfuly running EP-8.6.2 without these issues. Thank you for the quick response.

4

This could be a number of issues, so it's hard to tell without a little more information. How are you running Enterprise Search and Elasticsearch? Are they in a Docker setup, or running just on your local machine as binaries?

One suggestion that might work: have you tried using https://localhost:9200 as the elasticsearch.host in your enterprise-search.yml?

5

Are there any exceptions in the logs? Double check you have followed the SSL cert steps. Its a self signed cert thats created by Elasticsearch docker.

6

This is not a docker install. The is a on prem red hat server. Tring to get Elastic Search 8.6.2 and Enterprise Search 8.6.2 running on the same box. Elastic Search 8.6.2 is running no problems using curl commands have put-post-get get data
example return data I added
curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:password -X GET "https://localhost:9200/index-name/_search?pretty=true&q=*:*"
Have try localhost as suggested no change.
Is Kibana required for Enterprise Search 8.6.2 to run saw that as prerequisite somewhere Kibana not installed

7

Elastic search http_ca.crt was creates and works no problem with curl command.
I have this in the enterprise search yml file now
allow_es_settings_modification:true

elasticsearch.host:https://localhost:9200
elasticsearch.username:elastic
elasticsearch.password:password
elasticsearch.ssl.enabled:true
elasticsearch.ssl.certificate_authority:/etc/elasticsearch/certs/http_ca.crt

Still not working.
thank you for response

8

Yes, Kibana is required to be able to run Enterprise Search, see:

What is the exact error you're getting after these changes?

9

So enterprise should work without kibana?

In ep yml I have this
elasticsearch.ssl.certificate_authority: /etc/elasticsearch/certs/http_ca.crt

Getting this error

Invalid config file (/usr/share/enterprise-search/config/enterprise-search.yml):
The setting '#/elasticsearch/ssl/certificate_authority' is not valid: error reading file '/etc/elasticsearch/certs/http_ca.crt'

With CURL command works same path to http_ca.crt and user elastic : password

curl --cacert /etc/elasticsearch/certs/http_ca.crt -u elastic:<passwors> https://localhost:9200
{
  "name" : "rofccs901a",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "uS1qTEERSJmATh40KAV-Cw",
  "version" : {
    "number" : "8.6.2",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "2d58d0f136141f03239816a4e360a8d17b6d8f29",
    "build_date" : "2023-02-13T09:35:20.314882762Z",
    "build_snapshot" : false,
    "lucene_version" : "9.4.2",
    "minimum_wire_compatibility_version" : "7.17.0",
    "minimum_index_compatibility_version" : "7.0.0"
  },
  "tagline" : "You Know, for Search"
}
10

You do need to run Kibana to run Enterprise Search. It is a requirement.

To set up your certificate authority for Enterprise Search, you need to create a Java KeyStore. We’ve documented how to do that here:
https://www.elastic.co/guide/en/enterprise-search/current/configure-ssl-tls.html

11

ok I will setup Kibana

I do have a java key store and have it set in the yml file
ent_search.ssl.keystore.path: /usr/share/enterprise-search/config/JavaKeystrore.jks

So there must be an order to the start up/shut down
Startup
Elastic Search - Kibana - Enterprise Search
Shut down
Enterprise Search - Kibana - Elastic Search

12

Ok kibana installed and running good
Java keystore with a CA cert in it create per link up above
Error says
"Failed to connect to Elasticsearch backend. Make sure it is running and healthy."
Elasticsearch IS running and healthy and Enterprise yml has correct es user/pwd
kibana.yml lines added

enterpriseSearch.host: https://server.host.edu:3002
enterpriseSearch.ssl.verificationMode: certificate
enterpriseSearch.ssl.certificateAuthorities: /usr/share/enterprise-search/config/server_ca_w_chain.cer

Lines added to enterprise.yml

ent_search.ssl.enabled: true
ent_search.ssl.keystore.path: /usr/share/enterprise-search/config/server.jks
ent_search.ssl.keystore.password: password
ent_search.ssl.keystore.key_password: password
ent_search.external_url: https://server.host.edu:3002

ERROR MESSAGE

 "exception": "/usr/share/enterprise-search/lib/war/lib/middleware/request_logging_middleware.class:55: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (Faraday::ClientError)\n",
  "duration": 25.6,
  "stack": [

FULL ERROR MESSAGE

[2023-04-02T19:07:04.634+00:00][80952][4004][app-server][WARN]: [pre-flight] Failed to connect to Elasticsearch backend. Make sure it is running and healthy.
[2023-04-02T19:07:04.634+00:00][80952][4004][app-server][INFO]: [pre-flight] Error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:179: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (Faraday::ClientError)

[2023-04-02T19:07:09.667+00:00][80952][4004][es][DEBUG]: {
  "request": {
    "url": "https://localhost:9200/",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.10.2"
    },
    "params": null,
    "body": null
  },
  "exception": "/usr/share/enterprise-search/lib/war/lib/middleware/request_logging_middleware.class:55: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (Faraday::ClientError)\n",
  "duration": 25.6,
  "stack": [
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:651:in `block in get_raw'",
    "/usr/share/enterprise-search/lib/war/lib/apm_helpers.class:41:in `es_action_instrument'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:749:in `instrument'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch.class:650:in `get_raw'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:134:in `check_es_connection_with_retries!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:32:in `check_es_connection!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:19:in `block in run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:18:in `run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/elasticsearch_checks.class:14:in `run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:291:in `configure_elasticsearch!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:270:in `configure!'",
    "/usr/share/enterprise-search/lib/war/config/application.class:20:in `<main>'",
    "/usr/share/enterprise-search/lib/war/config/application.rb:1:in `<main>'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:36:in `initialize'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:148:in `run_supported_command'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:130:in `run_command'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
13

Hi @neil.maffitt, this error message tells me something is wrong with the keystore you generated:

PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target (Faraday::ClientError)

I can't help you there, other than to point at the documentation to create a Java keystore: Configure SSL/TLS | Enterprise Search documentation [8.11] | Elastic

You may want to try the keytool inside the Enterprise Search Docker image we provide, so we can be sure version mismatches aren't causing issues.

14

Because "PKIX path building" failed is the error I have been battling all along (see original post) I followed "Configure SSL/TLS | Elastic Enterprise Search documentation [8.7] | Elastic
[/quote]" document closely and documented step by step.
Steps:

Generate private key
   openssl genrsa -out filename.private.key 2048

Generate CSR file for SSL Certificate from setigo
   openssl req -new -key filename.private.key -out filename.csr -subj '/CN=data/OU=data IT/O=data/L=data/ST=data/C=data'

Upload CSR to CA

Download Certificate mv  (w/ chain), PEM encoded from CA
filenameu_ca_w_chain.cer

Generate p12 file with Certificate from CA
openssl pkcs12 -export -in filename_ca_w_chain.cer -inkey filename.private.key -out filename.p12 -name ent-search

Generate Java keystore file
keytool -importkeystore -destkeystore filename.jks -srckeystore filename.p12 -srcstoretype PKCS12 -storepass password -srcstorepass password

Optional view jks file
keytool -list -v -keystore rofccs901a.mayo.edu.jks
Alias name: ent-search
Creation date: Mar 30, 2023
Entry type: PrivateKeyEntry
Valid from: Wed Mar 29 19:00:00 CDT 2023 until: Fri Mar 29 18:59:59 CDT 2024
Certificate fingerprints:
         SHA1: 88:70:C7:0E:59:0E:AB:DE:D4:53:79:C0:42:AF:63:63:C1:C6:F0:91
         SHA256: EC:8D:93:E9:69:57:7A:99:A1:5C:8F:16:87:D5:DD:5A:2A:AA:32:E4:CE:63:FD:A8:8F:11:D2:40:17:83:25:C3
Signature algorithm name: SHA256withRSA
Subject Public Key Algorithm: 2048-bit RSA key

Files created and used
        filename.private.key
        filename.csr
        filename_ca_w_chain.cer
        filename.p12
        filename.jks

Please advise if you see a problem. Thanks

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.