Unexpected exception while running Enterprise Search: Error: Read timed out at

Nids_Rai · June 14, 2022, 7:35am

Getting read timeout error while setting up, Appsearch in ECK cluster. Elastic is up and running fine.

The LOG :

[2022-06-14T06:10:00.351+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Failed] Creating indices for 38 models: Error = Faraday::TimeoutError: Read timed out
[2022-06-14T06:10:00.530+00:00][7][2000][es][DEBUG]: {
"request": {
"url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
"method": "get",
"headers": {
"Authorization": "[FILTERED]",
"Content-Type": "application/json",
"x-elastic-product-origin": "enterprise-search",
"User-Agent": "Faraday v1.8.0"
},
"params": null,
"body": null
},
"response": {
"status": 200,
"headers": {
"x-elastic-product": "Elasticsearch",
"content-type": "application/json",
"content-length": "477"
},
"body": "{"_index":".ent-search-db-lock-20200304","_id":"installation","_version":120,"_seq_no":44197,"_primary_term":1,"found":true,"_source":{"product_version":"8.1.2","locked_at":"2022-06-14T06:08:14+00:00","expires_at":"2022-06-14T06:11:59+00:00","last_heartbeat_at":"2022-06-14T06:09:59+00:00","status_update_at":"2022-06-14T06:08:13+00:00","last_status":"[Starting] Creating indices for 38 models","node_name":"enterprise-search-bselastic-ent-bf4f99bbd-fjwgr","pid":7,"tid":2000}}"
},

apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-bselastic
spec:
  version: 8.1.2
  count: 1
  elasticsearchRef:
    name: bselastic
  config:
    log_level: debug
  podTemplate:
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: worker
      containers:
      - name: enterprise-search
        env:
        - name: JAVA_OPTS
          value: -Xms2g -Xmx2g
        - name: "elasticsearch.startup_retry.interval"
          value: "30"
        resources:
          limits:
            memory: "8Gi"
            cpu: 3
          requests:
            memory: "8Gi"
            cpu: 3

[root@1175014-kubemaster01]# kubectl -n bselastic  get pods
NAME                                               READY   STATUS    RESTARTS   AGE
bselastic-es-default-0                             1/1     Running   0          13d
bselastic-es-default-1                             1/1     Running   0          13d
bselastic-es-default-2                             1/1     Running   0          13d
enterprise-search-bselastic-ent-75c9546c9c-sg2x4   0/1     Running   0          12s

While indexing it does few and then it starts giving read time out error

Artem_Shelkovnikov · June 16, 2022, 11:59am

Hi @Nids_Rai!

Can you elaborate a bit more about your set up? Looking at the logs, it looks like there's a problem with connectivity between Enterprise Search and Elasticsearch.

I could think of problems with latency between Enterprise Search and Elasticsearch or overloaded Elasticsearch could lead to such state.

Can you somehow check the latency between Enterprise Search and Elasticsearch installations? Is your Elasticsearch instance heavily loaded?

Nids_Rai · June 17, 2022, 6:16am

Hi @Artem_Shelkovnikov ,
No. the Elasticsearch is not overloaded we are not doing anything over it apart from starting appsearch service.
I am using this yaml for Elasticsearch with one master and 3 data nodes.

---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: bselastic
spec:
  version: 8.1.2
  nodeSets:
  - name: masters
    count: 1
    config:
      node.roles: ["master",
#                  "data",
                  ]
      xpack.ml.enabled: true
      logger.org.elasticsearch: debug
#     elasticsearch.shardTimeout: 120
    podTemplate:
      spec:
        nodeSelector:
          # Select node to make sure `sysctl` command works.
          node-role.kubernetes.io/worker: worker
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms2g -Xmx2g
          resources:
            limits:
              memory: "3Gi"
              cpu: 4
            requests:
              memory: "3Gi"
              cpu: 4
#       initContainers:
#       - name: sysctl
#         resources:
#           limits:
#             memory: "50Mi"
#             cpu: "100m"
#           requests:
#             memory: "50Mi"
#             cpu: "100m"
#         securityContext:
#           privileged: true
#         command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
    # Volumeclaim needed to add volume, it was giving error for not volume claim
    # and its not starting pod.
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: standard
  - name: data-node
    count: 3
    config:
      node.roles: ["data", "ingest"]
      logger.org.elasticsearch: debug
#     elasticsearch.shardTimeout: 120
    podTemplate:
      spec:
        nodeSelector:
          node-role.kubernetes.io/worker: worker
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms2g -Xmx2g
          resources:
            limits:
              memory: "3Gi"
              cpu: 4
            requests:
              memory: "3Gi"
              cpu: 4
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: standard

and the Elasticsearch is running fine .

kubectl get elasticsearch -n bselastic
NAME HEALTH NODES VERSION PHASE AGE
bselastic yellow 3 8.2.2 Ready 16d

right after doing this we , We ran app search service . our elastic is not handing anything as of now . Apart from app search thing.

it starts with indexes .and then later breaks with timeout.

[2022-06-06T06:21:48.510+00:00][7][2000][app-server][INFO]: Elasticsearch looks healthy and configured correctly to run Enterprise Search
[2022-06-06T06:21:48.512+00:00][7][2000][app-server][INFO]: Performing pre-flight checks for Kibana running on http://localhost:5601...
[2022-06-06T06:21:48.535+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Kibana backend. Make sure it is running and healthy.
[2022-06-06T06:21:48.538+00:00][7][2000][app-server][ERROR]: Could not connect to Kibana backend after 0 seconds.
[2022-06-06T06:21:48.539+00:00][7][2000][app-server][WARN]: Enterprise Search is unable to connect to Kibana. Ensure it is running at http://localhost:5601 for user bselastic-enterprise-search-bselastic-ent-user.
[2022-06-06T06:21:51.271+00:00][7][2000][app-server][INFO]: Elastic APM agent is disabled
[2022-06-06T06:22:01.478+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Starting] Ensuring migrations tracking index exists
[2022-06-06T06:22:05.973+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Finished] Ensuring migrations tracking index exists
[2022-06-06T06:22:06.458+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Starting] Creating indices for 38 models
[2022-06-06T06:22:06.645+00:00][7][2000][app-server][INFO]: Creating index for Actastic::Schema: connectors_jobs_v5
[2022-06-06T06:22:06.668+00:00][7][2000][app-server][WARN]: Index .ent-search-actastic-connectors_jobs_v5 already exists!
[2022-06-06T06:22:11.914+00:00][7][2000][app-server][WARN]: Going to re-create the existing empty index .ent-search-actastic-connectors_jobs_v5
{"timestamp": "2022-06-06T06:22:14+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2022-06-06T06:22:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2022-06-06T06:22:30+00:00", "message": "readiness probe failed", "curl_rc": "7"}

Nids_Rai · June 17, 2022, 6:23am

The lastest last few logs :


[2022-06-17T06:21:27.695+00:00][7][2044][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-17T06:21:26+00:00\",\"expires_at\":\"2022-06-17T06:23:26+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Starting] Creating indices for 38 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "177"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":15,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":60722,\"_primary_term\":1}"
  },
  "duration": 691.0,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:10:in `block in update_lock'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
{"timestamp": "2022-06-17T06:21:30+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-17T06:21:32.701+00:00][7][2044][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 38 models
[2022-06-17T06:21:32.753+00:00][7][2044][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "477"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":15,\"_seq_no\":60722,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.1.2\",\"locked_at\":\"2022-06-17T06:20:24+00:00\",\"expires_at\":\"2022-06-17T06:23:26+00:00\",\"last_heartbeat_at\":\"2022-06-17T06:21:26+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Starting] Creating indices for 38 models\",\"node_name\":\"enterprise-search-bselastic-ent-75f9f87cb4-z4xsg\",\"pid\":7,\"tid\":2000}}"
  },
  "duration": 41.1,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:10:in `block in _retrieve_lock'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:10:in `block in update_lock'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
2022-06-17T06:22:19.177+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Shutting down the heartbeat timer...
[2022-06-17T06:22:19.178+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Waiting for heartbeat thread to finish...
[2022-06-17T06:22:19.273+00:00][7][2044][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-17T06:22:19+00:00\",\"expires_at\":\"2022-06-17T06:24:19+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Failed] Creating indices for 38 models: Error = Faraday::TimeoutError: Read timed out\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "177"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":25,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":60732,\"_primary_term\":1}"
  },
  "duration": 214.4,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:10:in `block in update_lock'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-17T06:22:19.280+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] The heartbeat should be down now
[2022-06-17T06:22:19.282+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Status: Releasing the lock
[2022-06-17T06:22:19.299+00:00][7][2000][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "522"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":25,\"_seq_no\":60732,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.1.2\",\"locked_at\":\"2022-06-17T06:20:24+00:00\",\"expires_at\":\"2022-06-17T06:24:19+00:00\",\"last_heartbeat_at\":\"2022-06-17T06:22:19+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Failed] Creating indices for 38 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-75f9f87cb4-z4xsg\",\"pid\":7,\"tid\":2000}}"
  },
  "duration": 6.6,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:466:in `release_lock'",
    "lib/apm_helpers.class:10:in `block in release_lock'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:399:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-17T06:22:19.946+00:00][7][2000][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation?if_primary_term=1&if_seq_no=60732",
    "method": "delete",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "177"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":26,\"result\":\"deleted\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":60733,\"_primary_term\":1}"
  },
  "duration": 634.0,
  "stack": [
    "lib/db_lock.class:305:in `delete_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:470:in `release_lock'",
    "lib/apm_helpers.class:10:in `block in release_lock'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:399:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-17T06:22:19.949+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Successfully released lock
Unexpected exception while running Enterprise Search:
Error: Read timed out at

Artem_Shelkovnikov · June 17, 2022, 2:50pm

Judging by this line, I see that Elasticsearch is not healthy - it's in Yellow status, so it's possible that part of its nodes are down or not available.

I suggest looking at Elasticsearch logs and checking the warning and error messages there - they can give you a hint what might be wrong with Elasticsearch.

I think when the cluster goes down to "green" state, the problem will go away by itself.

Nids_Rai · June 17, 2022, 5:46pm

yes, But when I started it it was green , Infact now it is green .

kubectl get Elasticsearch -n bselastic

NAME        HEALTH   NODES   VERSION   PHASE   AGE
bselastic   green    3       8.2.2     Ready   17d

The the pod status -
kubectl get pods -n bselastic

NAME                                               READY   STATUS             RESTARTS        AGE
[root@1175014-kubemaster01 ]# kubectl get pods -n bselastic
NAME                                               READY   STATUS    RESTARTS   AGE
bselastic-es-default-0                             1/1     Running   0          17d
bselastic-es-default-1                             1/1     Running   0          17d
bselastic-es-default-2                             1/1     Running   0          17d
enterprise-search-bselastic-ent-7cf8754f75-dljmd   0/1     Running   0          5s
kibana-bselastic-kb-56d5f544bb-68sf5               1/1     Running   0          16d

Also, Is there any way I can increase the timeout in enterprise search . I feel like it would be timeout issue .

Artem_Shelkovnikov · June 17, 2022, 6:22pm

I see, so Elasticsearch looks healthy.

I noticed from the logs is that your Elasticsearch has version 8.2.2 , but Enterprise Search is 8.1.2 . Is it intended? Normally, it’s not possible to run the stack with version incompatibility, though I wonder why there is no error message indicating this. I can suggest using the same version for both Elasticsearch and Enterprise Search deployments.

Another idea that I have is that Enterprise Search instance might not have enough resources to run - I would recommend allocating at least 2GB memory for Enterprise Search environment. Can you make sure that sufficient memory is used for the deployment?

Nids_Rai · June 22, 2022, 12:09pm

Hi @Artem_Shelkovnikov , I tried the same version and allocated more than 2GB of memory for the Enterprise Search environment.
but the issue persists. Also, I noticed that: the Elasticsearch is Green and as soon as I start the App search it remains green for a while and as soon as the log shows curl_7 error on app search,it turns to yellow.
Do we have any way to increase the timeout of indexing in appsearch ?
Do you have any sample setup or YAML of working app search which I can use as a reference( I have already referred the one at doc)?

still confused about this timeout .

 {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "522"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":24,\"_seq_no\":95602,\"_primary_term\":3,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-22T12:18:44+00:00\",\"expires_at\":\"2022-06-22T12:22:40+00:00\",\"last_heartbeat_at\":\"2022-06-22T12:20:40+00:00\",\"status_update_at\":\"2022-06-22T12:18:44+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-7469648db4-q9l94\",\"pid\":7,\"tid\":4004}}"
  },

Artem_Shelkovnikov · June 23, 2022, 12:36pm

Do we have any way to increase the timeout of indexing in appsearch ?

I don't think it's possible, there seems to be no such setting - and the problem even happens during installation, I don't think timeout increase will help here, unfortunately.

Also, I noticed that: the Elasticsearch is Green and as soon as I start the App search it remains green for a while and as soon as the log shows curl_7 error on app search,it turns to yellow.

I think it's our hint here. I think that your Enterprise Search is configured correctly, but something is happening with Elasticsearch. I think inspecting the logs of Elasticsearch during the moment when you run Enterprise Search will give us required information that will help understanding what happens.

Can you share Elasticsearch logs that are displayed once you start Enterprise Search against it? It should be some [warning] or [error] line that explains why the health of Elasticsearch changes.

maryna.cherniavska · June 23, 2022, 1:02pm

Nids_Rai:

{"timestamp": "2022-06-06T06:22:14+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2022-06-06T06:22:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2022-06-06T06:22:30+00:00", "message": "readiness probe failed", "curl_rc": "7"}

Hi @Nids_Rai. These last messages aren't coming from Elasticsearch or Enterprise Search - this is kubernetes pod readiness probe. I don't see liveness or readiness probe explicitly configured in the yaml that you provided for either Elasticsearch or Enterprise Search. It might help if you provided that part of the configuration yaml here. I am suspecting it might be an issue of Enterprise Search just not having enough time to start before the pod is killed due to readiness probe having failed. It might be a false hypothesis, but I'd still like to see that configuration.

Nids_Rai · June 27, 2022, 7:58am

Hi @maryna.cherniavska, we have not configured any "readiness probe" explicitly. This is coming from the operator itself.

maryna.cherniavska · June 28, 2022, 9:30am

Hi @Nids_Rai .

Could you try to give more time to the readiness probe? It's taken from an env variable as shown here in the docs. So you could do this in your spec:

env:
- name: READINESS_PROBE_TIMEOUT
  value: "120"

And see if that gives you more time and if logs go further?

Nids_Rai · June 28, 2022, 12:32pm

Hi @maryna.cherniavska
I am using this yaml and have already configured timeout .

apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-bselastic
spec:
  version: 8.2.2
  count: 1
  elasticsearchRef:
    name: bselastic
  config:
    log_level: debug
  podTemplate:
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: worker
      containers:
      - name: enterprise-search
        env:
        - name: JAVA_OPTS
          value: -Xms2g -Xmx2g
        - name: "elasticsearch.startup_retry.interval"
          value: "30"
        - name: READINESS_PROBE_TIMEOUT
          value: "500"
        resources:
          limits:
            memory: "2Gi"
            cpu: 3
          requests:
            memory: "2Gi"
            cpu: 3

Even while Describe it shows that env as 500 and still getting curl_7

[root@1175014-kubemaster01 nilesh]# kubectl describe pod enterprise-search-bselastic-ent-5668b4f75b-89m8r -n bselastic
Name:         enterprise-search-bselastic-ent-5668b4f75b-89m8r
Namespace:    bselastic
Priority:     0
Node:         1175018-kubeworker03.sb.rackspace.com/172.17.201.225
Start Time:   Tue, 28 Jun 2022 12:30:14 +0000
Labels:       common.k8s.elastic.co/type=enterprise-search
              enterprisesearch.k8s.elastic.co/name=enterprise-search-bselastic
              enterprisesearch.k8s.elastic.co/version=8.2.2
              pod-template-hash=5668b4f75b
Annotations:  cni.projectcalico.org/containerID: cfcb93ef4a329c0aa05310c9a2b69682643508ac161181b87508b270fc333937
              cni.projectcalico.org/podIP: 192.168.63.209/32
              cni.projectcalico.org/podIPs: 192.168.63.209/32
              enterprisesearch.k8s.elastic.co/config-hash: 3190656219
Status:       Running
IP:           192.168.63.209
IPs:
  IP:           192.168.63.209
Controlled By:  ReplicaSet/enterprise-search-bselastic-ent-5668b4f75b
Containers:
  enterprise-search:
    Container ID:   containerd://879f634bb1bce1e78fe1b589f18373cc5aaa0d91217baef9be1d5f321578f569
    Image:          docker.elastic.co/enterprise-search/enterprise-search:8.2.2
    Image ID:       docker.elastic.co/enterprise-search/enterprise-search@sha256:672d9cee9c5c80fe5540c1f0925ae271cf781f1ef237109578a258452fd948dd
    Port:           3002/TCP
    Host Port:      0/TCP
    State:          Running
      Started:      Tue, 28 Jun 2022 12:30:15 +0000
    Ready:          False
    Restart Count:  0
    Limits:
      cpu:     3
      memory:  2Gi
    Requests:
      cpu:      3
      memory:   2Gi
    Readiness:  exec [bash /mnt/elastic-internal/scripts/readiness-probe.sh] delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:
      JAVA_OPTS:                                           -Xms2g -Xmx2g
      elasticsearch.startup_retry.interval:                30
      READINESS_PROBE_TIMEOUT:                             500
      NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME:          IAD.Prod
      NEW_RELIC_METADATA_KUBERNETES_NODE_NAME:              (v1:spec.nodeName)
      NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME:        bselastic (v1:metadata.namespace)
      NEW_RELIC_METADATA_KUBERNETES_POD_NAME:              enterprise-search-bselastic-ent-5668b4f75b-89m8r (v1:metadata.name)
      NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME:        enterprise-search
      NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME:  docker.elastic.co/enterprise-search/enterprise-search:8.2.2
      NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME:       enterprise-search-bselastic-ent
    Mounts:
      /mnt/elastic-internal/es-certs from es-certs (ro)
      /mnt/elastic-internal/http-certs from elastic-internal-http-certificates (ro)
      /mnt/elastic-internal/scripts/readiness-probe.sh from readiness-probe (ro,path="readiness-probe.sh")
      /usr/share/enterprise-search/config/enterprise-search.yml from config (ro,path="enterprise-search.yml")
      /var/log/enterprise-search from logs (rw)
Conditions:
  Type              Status
  Initialized       True
  Ready             False
  ContainersReady   False
  PodScheduled      True
Volumes:
  config:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  enterprise-search-bselastic-ent-config
    Optional:    false
  elastic-internal-http-certificates:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  enterprise-search-bselastic-ent-http-certs-internal
    Optional:    false
  es-certs:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  enterprise-search-bselastic-ent-es-ca
    Optional:    false
  logs:
    Type:       EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit:  <unset>
  readiness-probe:
    Type:        Secret (a volume populated by a Secret)
    SecretName:  enterprise-search-bselastic-ent-config
    Optional:    false
QoS Class:       Guaranteed
Node-Selectors:  node-role.kubernetes.io/worker=worker
Tolerations:     node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type     Reason     Age              From               Message
  ----     ------     ----             ----               -------
  Normal   Scheduled  96s              default-scheduler  Successfully assigned bselastic/enterprise-search-bselastic-ent-5668b4f75b-89m8r to 1175018-kubeworker03.sb.rackspace.com
  Normal   Pulled     96s              kubelet            Container image "docker.elastic.co/enterprise-search/enterprise-search:8.2.2" already present on machine
  Normal   Created    96s              kubelet            Created container enterprise-search
  Normal   Started    95s              kubelet            Started container enterprise-search
  Warning  Unhealthy  26s              kubelet            Readiness probe failed: {"timestamp": "2022-06-28T12:31:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
  Warning  Unhealthy  16s              kubelet            Readiness probe failed: {"timestamp": "2022-06-28T12:31:34+00:00", "message": "readiness probe failed", "curl_rc": "7"}
  Warning  Unhealthy  6s (x2 over 6s)  kubelet            Readiness probe failed: {"timestamp": "2022-06-28T12:31:44+00:00", "message": "readiness probe failed", "curl_rc": "7"}

Nids_Rai · June 28, 2022, 12:49pm

and the latest few logs are -


[2022-06-28T12:42:15.216+00:00][8][4004][app-server][INFO]: Creating index for Actastic::Schema: workplace_search_pre_content_sources_v3
[2022-06-28T12:42:17.484+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:17.531+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":172,\"_seq_no\":171,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:11+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:11+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 43.1,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:18.030+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:17+00:00\",\"expires_at\":\"2022-06-28T12:44:17+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":173,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":172,\"_primary_term\":1}"
  },
  "duration": 493.1,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:23.034+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:23.083+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":173,\"_seq_no\":172,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:17+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:17+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 45.4,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
{"timestamp": "2022-06-28T12:42:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-28T12:42:24.204+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:23+00:00\",\"expires_at\":\"2022-06-28T12:44:23+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":174,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":173,\"_primary_term\":1}"
  },
  "duration": 1115.1,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:29.208+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:29.351+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":174,\"_seq_no\":173,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:23+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:23+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 138.6,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:29.530+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:29+00:00\",\"expires_at\":\"2022-06-28T12:44:29+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":175,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":174,\"_primary_term\":1}"
  },
  "duration": 172.7,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
{"timestamp": "2022-06-28T12:42:34+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-28T12:42:34.533+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:34.581+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":175,\"_seq_no\":174,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:29+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:29+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 43.7,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:35.581+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:34+00:00\",\"expires_at\":\"2022-06-28T12:44:34+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":176,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":175,\"_primary_term\":1}"
  },
  "duration": 994.0,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:40.585+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:40.677+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":176,\"_seq_no\":175,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:34+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:34+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 85.9,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:41.111+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:40+00:00\",\"expires_at\":\"2022-06-28T12:44:40+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":177,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":176,\"_primary_term\":1}"
  },
  "duration": 426.5,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:42.004+00:00][8][4188][app-server][DEBUG]: Summarizing metrics...
[2022-06-28T12:42:42.004+00:00][8][4188][app-server][DEBUG]: Done summarizing 0 metrics.
{"timestamp": "2022-06-28T12:42:44+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-28T12:42:45.254+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-actastic-workplace_search_pre_content_sources_v3",
    "method": "put",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"settings\":{\"index\":{\"hidden\":true,\"refresh_interval\":-1},\"number_of_shards\":1,\"auto_expand_replicas\":\"0-3\",\"priority\":250},\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"id\":{\"type\":\"keyword\"},\"created_at\":{\"type\":\"date\"},\"updated_at\":{\"type\":\"date\"},\"service_type\":{\"type\":\"keyword\"},\"frito_pie_account_id\":{\"type\":\"keyword\"},\"context\":{\"type\":\"keyword\"},\"configuration\":{\"type\":\"object\",\"enabled\":false}}},\"aliases\":{}}"
  },
  "exception": "/usr/share/enterprise-search/lib/war/lib/swiftype/es/index.class:59: Read timed out (Faraday::TimeoutError)\n",
  "duration": 30030.5,
  "stack": [
    "lib/actastic/schema.class:172:in `create_index!'",
    "lib/actastic/schema.class:195:in `create_index_and_mapping!'",
    "shared_togo/lib/shared_togo.class:912:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:910:in `block in each'",
    "shared_togo/lib/shared_togo.class:910:in `block in apply_actastic_migrations'",
    "lib/db_lock.class:182:in `with_status'",
    "shared_togo/lib/shared_togo.class:909:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo/audit_event.class:48:in `log_event_with_message'",
    "shared_togo/lib/shared_togo.class:907:in `apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:420:in `block in install!'",
    "lib/db_lock.class:171:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:45.255+00:00][8][4004][app-server][INFO]: [db_lock] [installation] Status: [Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out
[2022-06-28T12:42:45.307+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":177,\"_seq_no\":176,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:40+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:40+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 46.3,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:194:in `set_status'",
    "lib/db_lock.class:185:in `with_status'",
    "shared_togo/lib/shared_togo.class:909:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo/audit_event.class:48:in `log_event_with_message'",
    "shared_togo/lib/shared_togo.class:907:in `apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:420:in `block in install!'",
    "lib/db_lock.class:171:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.116+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out
[2022-06-28T12:42:46.149+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "521"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":178,\"_seq_no\":177,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:45+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:45+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 27.7,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:46.177+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:45+00:00\",\"expires_at\":\"2022-06-28T12:44:45+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":178,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":177,\"_primary_term\":1}"
  },
  "duration": 863.8,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:194:in `set_status'",
    "lib/db_lock.class:185:in `with_status'",
    "shared_togo/lib/shared_togo.class:909:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo/audit_event.class:48:in `log_event_with_message'",
    "shared_togo/lib/shared_togo.class:907:in `apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:420:in `block in install!'",
    "lib/db_lock.class:171:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.179+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Shutting down the heartbeat timer...
[2022-06-28T12:42:46.179+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Waiting for heartbeat thread to finish...
[2022-06-28T12:42:46.243+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:46+00:00\",\"expires_at\":\"2022-06-28T12:44:46+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":179,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":178,\"_primary_term\":1}"
  },
  "duration": 88.2,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:46.281+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] The heartbeat should be down now
[2022-06-28T12:42:46.282+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Status: Releasing the lock
[2022-06-28T12:42:46.301+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "521"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":179,\"_seq_no\":178,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:46+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:46+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 13.3,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:466:in `release_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.389+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation?if_primary_term=1&if_seq_no=178",
    "method": "delete",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":180,\"result\":\"deleted\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":179,\"_primary_term\":1}"
  },
  "duration": 80.4,
  "stack": [
    "lib/db_lock.class:305:in `delete_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:470:in `release_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.390+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Successfully released lock
Unexpected exception while running Enterprise Search:
Error: Read timed out at

Nids_Rai · June 28, 2022, 1:40pm

Hi @Artem_Shelkovnikov and @maryna.cherniavska , Do you guys have any working YAML for enterprise search which I can refer to.

Rich_Kuzsma · June 30, 2022, 9:56pm

Not sure if it helps, but I've tested Enterprise Search 8.1 using this YML file in a local single node development configuration: Elastic Enterprise Search ECK yaml configuration files for testing · GitHub

TWR · July 1, 2022, 12:37am

I just upgraded my on-prem implementation to 8.3 and have been seeing these Faraday errors in the logs since then. Did anyone ever discover a root cause for this?

Nids_Rai · July 4, 2022, 10:47am

Hi @Rich_Kuzsma I changed my version to 8.1 as mentioned in the file but the error is same .

    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation?if_primary_term=1&if_seq_no=85",
    "method": "delete",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "174"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":87,\"result\":\"deleted\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":86,\"_primary_term\":1}"
  },
  "duration": 297.6,
  "stack": [
    "lib/db_lock.class:305:in `delete_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:470:in `release_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:399:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-07-04T10:29:54.439+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Successfully released lock
Unexpected exception while running Enterprise Search:

Artem_Shelkovnikov · July 4, 2022, 2:39pm

@Nids_Rai @TWR can you share the logs of the Elasticsearch that correspond the moment when Enterprise Search starts?

At this point it looks like there is some problem with Elasticsearch configuration - might be for example connected to the lack of physical space on disk to store new indices that are generated when Enterprise Search starts.

TWR · July 5, 2022, 11:26am

In my case, I have two separate 5-node clusters, each with its own two-node Enterprise Search instance, and only one of them has this problem.

Both clusters were just deployed to newly-provisioned 5 VMs, so each cluster has a full terabyte of storage available.

Before I dive into the logs, would it be worthwhile to disable this setting in enterprise-search.yml:

allow_es_settings_modification

Enterprise Search needs one-time permission to alter Elasticsearch settings. Ensure the Elasticsearch settings are correct, then set the following to true. Or, adjust Elasticsearch’s config/elasticsearch.yml instead.

allow_es_settings_modification: false

The documentation Configuration | Elastic Enterprise Search Documentation [8.3] | Elastic says to set this to true OR adjust Elasticsearch's config instead, but it doesn't indicate what we would do in the elasticearch.yml instead...