ECK Enterprise Search: Pre-flight check with Kibana connection refused

Elastic Search
1

We have a requirement to use the web crawler to ingest pages from our organisation's own websites.

We currently connect to Elasticsearch v7.17 hosted on IBM Cloud with Enterprise Search version 7.17 and Kibana version 7.17 running via ECK version 2.11 on an Openshift Cluster at version 4.12.
We are able to access Kibana via route/service to Kibana UI and also access Enterprise Search via route/service to Enterprise Search UI. We can run the web crawler in this environment.

This has been working fine for us. However, we must now move to version 8.10 of Elasticsearch and we understand that version 8.10 requires a different configuration for Enterprise Search as the Enterprise Search UI is now to be accessed via Kibana. To do this we understand that Enterprise Search must connect with Kibana and Kibana must connect with Enterprise search.

We created a new environment where we are running Kibana v8.10.1 and Enterprise Search v8.10.1 via ECK Operator. Both pods connect with Elasticsearch v8.10 hosted on IBM Cloud but unfortunately we see errors with these Enterprise Search connecting to Kibana and Kibana connecting to Enterprise Search.

Enterprise Search pod logs show connection refused as follows:

[2024-01-29T15:27:02.388+00:00][7][4004][app-server][INFO]: Elasticsearch will be used for authentication
[2024-01-29T15:27:02.389+00:00][7][4004][app-server][INFO]: Elasticsearch looks healthy and configured correctly to run Enterprise Search
[2024-01-29T15:27:02.391+00:00][7][4004][app-server][INFO]: Performing pre-flight checks for Kibana running on https://kibana-ibm-nft-kb-http.universal-search-sit.svc:5601...
[2024-01-29T15:27:03.491+00:00][7][4004][es][DEBUG]: {
  "request": {
    "url": "https://kibana-ibm-nft-kb-http.universal-search-sit.svc:5601/api/status",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "User-Agent": "Faraday v1.10.3"
    },
    "params": null,
    "body": null
  },
  "exception": "/usr/share/enterprise-search/lib/war/lib/middleware/request_logging_middleware.class:56: Connect to kibana-ibm-nft-kb-http.universal-search-sit.svc:5601 [kibana-ibm-nft-kb-http.universal-search-sit.svc/172.21.101.131] failed: Connection refused (Connection refused) (Faraday::ConnectionFailed)\n",
  "duration": 1062.3,
  "stack": [
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:79:in `check_kibana_connection_with_retries!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:34:in `check_kibana_connection!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:18:in `block in run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:17:in `run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:13:in `run!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:313:in `configure_kibana!'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo.class:271:in `configure!'",
    "/usr/share/enterprise-search/lib/war/config/application.class:21:in `<main>'",
    "/usr/share/enterprise-search/lib/war/config/application.rb:1:in `<main>'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:36:in `initialize'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:148:in `run_supported_command'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:130:in `run_command'",
    "/usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2024-01-29T15:27:03.493+00:00][7][4004][app-server][WARN]: Failed to connect to Kibana backend. Make sure it is running and healthy.
[2024-01-29T15:27:03.504+00:00][7][4004][app-server][DEBUG]: Kibana connection error: /usr/share/enterprise-search/lib/war/shared_togo/lib/shared_togo/kibana_checks.class:128: Connect to kibana-ibm-nft-kb-http.universal-search-sit.svc:5601 [kibana-ibm-nft-kb-http.universal-search-sit.svc/172.21.101.131] failed: Connection refused (Connection refused) (Faraday::ConnectionFailed)

[2024-01-29T15:27:03.506+00:00][7][4004][app-server][ERROR]: Could not connect to Kibana backend after 1 seconds.
[2024-01-29T15:27:03.507+00:00][7][4004][app-server][WARN]: Enterprise Search is unable to connect to Kibana. Ensure it is running at https://kibana-ibm-nft-kb-http.universal-search-sit.svc:5601 for user admin.

The Kibana UI shows 502 Bad Gateway Error when we access Search folder and we see an error in the Kibana pod log as follows around about the same time.

* Jan 31 15:43:30 kibana-ibm-nft-kb-665b9cb6fb-b9rrt kibana ERROR [plugins.enterpriseSearch] Could not perform access check to Enterprise Search: TypeError [ERR_INVALID_PROTOCOL]: Protocol "https:" not supported. Expected "http:"

Our Enterprise Search yaml is as follows....

apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-nft
spec:
  config:
    elasticsearch.host: >-
      https://admin:<admin password>@<ib, elasticsearch url>
    elasticsearch.password: <admin password>
    elasticsearch.ssl.certificate_authority: /etc/certs/ca.crt
    elasticsearch.ssl.enabled: true
    elasticsearch.username: admin
    ent_search.external_url: 'https://enterprise-search-nft-ent-http.universal-search-sit.svc:3002'
    kibana.host: 'https://kibana-ibm-nft-kb-http.universal-search-sit.svc:5601'
    log_level: debug
  count: 1
  podTemplate:
    annotations:
      sidecar.istio.io/inject: 'true'
    spec:
      containers:
        - name: enterprise-search
          volumeMounts:
            - mountPath: /etc/certs
              name: elasticsearch-certs
              readOnly: true
            - mountPath: /etc/certs/kibana
              name: kibana-certs
              readOnly: true
      volumes:
        - name: elasticsearch-certs
          secret:
            secretName: ibm-elasticsearch-nft
        - name: kibana-certs
          secret:
            secretName: kibana-ibm-nft-kb-http-certs-internal
  version: 8.10.1

Our Kibana yaml is as follows....

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-ibm-nft
spec:
  config:
    elasticsearch.hosts:
      - >-
        https://admin:<admin password>@<ibm elastic search url>:30032
    elasticsearch.ssl.certificateAuthorities: /etc/certs/ca.crt
    elasticsearch.username: admin
    enterpriseSearch.host: 'https://enterprise-search-nft-ent-http.universal-search-sit.svc:3002'
    enterpriseSearch.ssl.certificateAuthorities: mnt/elastic-internal/enterprise-search-certs/ca.crt
    enterpriseSearch.ssl.verificationMode: certificate
    logging.root.level: debug
  count: 1
  podTemplate:
    spec:
      containers:
        - name: kibana
          volumeMounts:
            - mountPath: /etc/certs
              name: elasticsearch-certs
              readOnly: true
            - mountPath: /mnt/elastic-internal/enterprise-search-certs
              name: elastic-internal-enterprise-search-http-certificates
              readOnly: true
      volumes:
        - name: elasticsearch-certs
          secret:
            secretName: ibm-elasticsearch-nft
        - name: elastic-internal-enterprise-search-http-certificates
          secret:
            secretName: enterprise-search-nft-ent-http-certs-internal
  secureSettings:
    - secretName: kibana-nft-elasticsearch-credentials
  version: 8.10.1
status:
  availableNodes: 1
  count: 1
  health: green
  observedGeneration: 15
  selector: 'common.k8s.elastic.co/type=kibana,kibana.k8s.elastic.co/name=kibana-ibm-nft'
  version: 8.10.1

We are able to use curl to access kibana from the enterprise search pod terminal window where ca.crt is the kibana CA with CN=kibana-ibm-nft-http

curl https://kibana-ibm-nft-kb-http.universal-search-sit.svc:5601/api/status --cacert ca.crt

We are able to access Enterprise Search from kibana pod terminal window where ca.crt is the enterprise search CA with CN=enterprise-search-nft-http

curl https://admin:<admin password>@enterprise-search-nft-ent-http.universal-search-sit.svc:3003/api/ent/vat/internal/version

Any help with this woudl be greatly appreciated

Thanks in advance
John

2

We managed to figure it out and fix the problem. Posting our solution here hoping it will be useful to someone else. We are focussing on the connection between Kibana and Enterprise Search so not providing any details on the connections between Enterprise Search/ Kibana and the Elasticsearch database.

We had previously attempted to create instance of Enterprise Search named enterprise-search-nft-http and an instance of Kibana named kibana-ibm-nft so ECK had already created things like services and certificates named after the names we gave to these instances. The names that appear below are based on the names we gave to Enterprise Search and Kibana instances. You would have your names.

We found the Enterprise Search CA (CN=enterprise-search-nft-http) within the enterprise-search-nft-ent-http-certs-internal secret's ca.crt property.

We created a new public re-encrypt route linking to the enterprise search service which had been created by ECK.

kind: Route
apiVersion: route.openshift.io/v1
metadata:
  name: enterprise-search-nft
  labels:
    common.k8s.elastic.co/type: enterprise-search
    enterprisesearch.k8s.elastic.co/name: enterprise-search-nft
  annotations:
    openshift.io/host.generated: 'true'
spec:
  host: >-
    <our host name>
  to:
    kind: Service
    name: enterprise-search-nft-ent-http
    weight: 100
  port:
    targetPort: https
  tls:
    termination: reencrypt
    destinationCACertificate: |-
   <Certificate CN=enterprise-search-nft-http goes here>
    insecureEdgeTerminationPolicy: Allow
  wildcardPolicy: None

Our ACL permits traffic to this route on port 443 from our internal network. The route will present the cluster's default public certificate which in our case is a LetsEncrypt certificate so the route is both accessible and trusted by browsers within our network.

We deployed Enterprise Search from ECK using the following yaml.

apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-nft
spec:
  config:
    kibana.host: 'https://kibana-ibm-nft-kb-http.universal-search-sit.svc:5601'
    elasticsearch.username: admin
    elasticsearch.ssl.certificate_authority: /etc/certs/ca.crt
    log_level: debug
    elasticsearch.password: 
<elastic search password goes here>
    ent_search.external_url: >-
      <url of public route to Enterprise Search>
    elasticsearch.ssl.enabled: true
    elasticsearch.host: >-
      <url of IBM hosted Elasticsearch>
    skip_read_only_check: false
  count: 1
  podTemplate:
    spec:
      containers:
        - name: enterprise-search
          volumeMounts:
            - mountPath: /etc/certs
              name: elasticsearch-certs
              readOnly: true
  version: 8.10.1

Notes

  • ent_search.external_url: references the URL of the public route to Enterprise Search created above
  • kibana.host: references the Kibana service created by ECK.

We deployed Kibana from ECK using the following yaml.

apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
  name: kibana-ibm-nft
spec:
  config:
    elasticsearch.hosts:
      - >-
        <elastic search url goes here>
    elasticsearch.ssl.certificateAuthorities: /etc/certs/ca.crt
    elasticsearch.username: admin
    enterpriseSearch.host: >-
      https://enterprise-search-nft-ent-http.universal-search-sit.svc.cluster.local:3002
    enterpriseSearch.ssl.certificateAuthorities: /mnt/elastic-internal/enterprise-search-certs/ca.crt
    enterpriseSearch.ssl.verificationMode: certificate
    logging.root.level: debug
  count: 1
  podTemplate:
    spec:
      containers:
        - name: kibana
          volumeMounts:
            - mountPath: /etc/certs
              name: elasticsearch-certs
              readOnly: true
            - mountPath: /mnt/elastic-internal/enterprise-search-certs
              name: elastic-internal-enterprise-search-http-certificates
              readOnly: true
      volumes:
        - name: elasticsearch-certs
          secret:
            secretName: ibm-elasticsearch-nft
        - name: elastic-internal-enterprise-search-http-certificates
          secret:
            secretName: enterprise-search-nft-ent-http-certs-internal
  secureSettings:
    - secretName: kibana-nft-elasticsearch-credentials
  version: 8.10.1

Notes:

  • enterpriseSearch.host: https://enterprise-search-nft-ent-http.universal-search-sit.svc.cluster.local:3002 links with Enterprise Search directly via the Enterprise Search service created by ECK.
  • mount the enterprise search CA (CN=enterprise-search-nft-http) to /mnt/elastic-internal/enterprise-search-certs from elastic-internal-enterprise-search-http-certificates secret
  • enterpriseSearch.ssl.certificateAuthorities: references the enterprise search CA mounted to /mnt/elastic-internal/enterprise-search-certs/ca.crt

Once we had done this the 502 Gateway Error no longer displayed within Kibana and options such as Search Web Crawler became available within Kibana.

I hope someone finds this useful.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.