# Unexpected exception while running Enterprise Search: Error: Read timed out at

**URL:** <https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102>\
**Category:** Elastic Search\
**Created:** [June 14, 2022, 7:35am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102 "2022-06-14T07:35:20Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 14, 2022, 7:35am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/1 "2022-06-14T07:35:20Z")

</div>

Getting read timeout error while setting up, Appsearch in ECK cluster. Elastic is up and running fine.

The LOG :

> [2022-06-14T06:10:00.351+00:00][7][2000][app-server][INFO]: [db\_lock] [installation] Status: [Failed] Creating indices for 38 models: Error = Faraday::TimeoutError: Read timed out  
> [2022-06-14T06:10:00.530+00:00][7][2000][es][DEBUG]: {  
> "request": {  
> "url": "[https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/\_doc/installation](https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation)",  
> "method": "get",  
> "headers": {  
> "Authorization": "[FILTERED]",  
> "Content-Type": "application/json",  
> "x-elastic-product-origin": "enterprise-search",  
> "User-Agent": "Faraday v1.8.0"  
> },  
> "params": null,  
> "body": null  
> },  
> "response": {  
> "status": 200,  
> "headers": {  
> "x-elastic-product": "Elasticsearch",  
> "content-type": "application/json",  
> "content-length": "477"  
> },  
> "body": "{"\_index":".ent-search-db-lock-20200304","\_id":"installation","\_version":120,"\_seq\_no":44197,"\_primary\_term":1,"found":true,"\_source":{"product\_version":"8.1.2","locked\_at":"2022-06-14T06:08:14+00:00","expires\_at":"2022-06-14T06:11:59+00:00","last\_heartbeat\_at":"2022-06-14T06:09:59+00:00","status\_update\_at":"2022-06-14T06:08:13+00:00","last\_status":"[Starting] Creating indices for 38 models","node\_name":"enterprise-search-bselastic-ent-bf4f99bbd-fjwgr","pid":7,"tid":2000}}"  
> },

```auto
apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-bselastic
spec:
  version: 8.1.2
  count: 1
  elasticsearchRef:
    name: bselastic
  config:
    log_level: debug
  podTemplate:
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: worker
      containers:
      - name: enterprise-search
        env:
        - name: JAVA_OPTS
          value: -Xms2g -Xmx2g
        - name: "elasticsearch.startup_retry.interval"
          value: "30"
        resources:
          limits:
            memory: "8Gi"
            cpu: 3
          requests:
            memory: "8Gi"
            cpu: 3

```

```auto
[root@1175014-kubemaster01]# kubectl -n bselastic get pods
NAME READY STATUS RESTARTS AGE
bselastic-es-default-0 1/1 Running 0 13d
bselastic-es-default-1 1/1 Running 0 13d
bselastic-es-default-2 1/1 Running 0 13d
enterprise-search-bselastic-ent-75c9546c9c-sg2x4 0/1 Running 0 12s

```

**While indexing it does few and then it starts giving read time out error**

---

<div class="post-metadata">

**Author:** ![Artem\_Shelkovnikov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artem_shelkovnikov/32/134322_2.png) [@Artem\_Shelkovnikov](https://discuss.elastic.co/u/Artem_Shelkovnikov)\
**Post date:** [June 16, 2022, 11:59am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/2 "2022-06-16T11:59:06Z")

</div>

Hi @Nids_Rai!

Can you elaborate a bit more about your set up? Looking at the logs, it looks like there's a problem with connectivity between Enterprise Search and Elasticsearch.

I could think of problems with latency between Enterprise Search and Elasticsearch or overloaded Elasticsearch could lead to such state.

Can you somehow check the latency between Enterprise Search and Elasticsearch installations? Is your Elasticsearch instance heavily loaded?

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 17, 2022, 6:16am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/3 "2022-06-17T06:16:58Z")

</div>

Hi @Artem_Shelkovnikov ,  
No. the Elasticsearch is not overloaded we are not doing anything over it apart from starting appsearch service.  
I am using this yaml for Elasticsearch with one master and 3 data nodes.

```auto
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
  name: bselastic
spec:
  version: 8.1.2
  nodeSets:
  - name: masters
    count: 1
    config:
      node.roles: ["master",
# "data",
                  ]
      xpack.ml.enabled: true
      logger.org.elasticsearch: debug
# elasticsearch.shardTimeout: 120
    podTemplate:
      spec:
        nodeSelector:
          # Select node to make sure `sysctl` command works.
          node-role.kubernetes.io/worker: worker
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms2g -Xmx2g
          resources:
            limits:
              memory: "3Gi"
              cpu: 4
            requests:
              memory: "3Gi"
              cpu: 4
# initContainers:
# - name: sysctl
# resources:
# limits:
# memory: "50Mi"
# cpu: "100m"
# requests:
# memory: "50Mi"
# cpu: "100m"
# securityContext:
# privileged: true
# command: ['sh', '-c', 'sysctl -w vm.max_map_count=262144']
    # Volumeclaim needed to add volume, it was giving error for not volume claim
    # and its not starting pod.
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data # Do not change this name unless you set up a volume mount for the data path.
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: standard
  - name: data-node
    count: 3
    config:
      node.roles: ["data", "ingest"]
      logger.org.elasticsearch: debug
# elasticsearch.shardTimeout: 120
    podTemplate:
      spec:
        nodeSelector:
          node-role.kubernetes.io/worker: worker
        containers:
        - name: elasticsearch
          env:
          - name: ES_JAVA_OPTS
            value: -Xms2g -Xmx2g
          resources:
            limits:
              memory: "3Gi"
              cpu: 4
            requests:
              memory: "3Gi"
              cpu: 4
    volumeClaimTemplates:
    - metadata:
        name: elasticsearch-data
      spec:
        accessModes:
        - ReadWriteOnce
        resources:
          requests:
            storage: 5Gi
        storageClassName: standard

```

and the Elasticsearch is running fine .

> kubectl get elasticsearch -n bselastic  
> NAME HEALTH NODES VERSION PHASE AGE  
> bselastic yellow 3 8.2.2 Ready 16d

right after doing this we , We ran app search service . our elastic is not handing anything as of now . Apart from app search thing.

it starts with indexes .and then later breaks with timeout.

```auto
[2022-06-06T06:21:48.510+00:00][7][2000][app-server][INFO]: Elasticsearch looks healthy and configured correctly to run Enterprise Search
[2022-06-06T06:21:48.512+00:00][7][2000][app-server][INFO]: Performing pre-flight checks for Kibana running on http://localhost:5601...
[2022-06-06T06:21:48.535+00:00][7][2000][app-server][WARN]: [pre-flight] Failed to connect to Kibana backend. Make sure it is running and healthy.
[2022-06-06T06:21:48.538+00:00][7][2000][app-server][ERROR]: Could not connect to Kibana backend after 0 seconds.
[2022-06-06T06:21:48.539+00:00][7][2000][app-server][WARN]: Enterprise Search is unable to connect to Kibana. Ensure it is running at http://localhost:5601 for user bselastic-enterprise-search-bselastic-ent-user.
[2022-06-06T06:21:51.271+00:00][7][2000][app-server][INFO]: Elastic APM agent is disabled
[2022-06-06T06:22:01.478+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Starting] Ensuring migrations tracking index exists
[2022-06-06T06:22:05.973+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Finished] Ensuring migrations tracking index exists
[2022-06-06T06:22:06.458+00:00][7][2000][app-server][INFO]: [db_lock] [installation] Status: [Starting] Creating indices for 38 models
[2022-06-06T06:22:06.645+00:00][7][2000][app-server][INFO]: Creating index for Actastic::Schema: connectors_jobs_v5
[2022-06-06T06:22:06.668+00:00][7][2000][app-server][WARN]: Index .ent-search-actastic-connectors_jobs_v5 already exists!
[2022-06-06T06:22:11.914+00:00][7][2000][app-server][WARN]: Going to re-create the existing empty index .ent-search-actastic-connectors_jobs_v5
{"timestamp": "2022-06-06T06:22:14+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2022-06-06T06:22:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
{"timestamp": "2022-06-06T06:22:30+00:00", "message": "readiness probe failed", "curl_rc": "7"}

```

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 17, 2022, 6:23am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/4 "2022-06-17T06:23:47Z")

</div>

The lastest last few logs :

```auto

[2022-06-17T06:21:27.695+00:00][7][2044][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-17T06:21:26+00:00\",\"expires_at\":\"2022-06-17T06:23:26+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Starting] Creating indices for 38 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "177"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":15,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":60722,\"_primary_term\":1}"
  },
  "duration": 691.0,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:10:in `block in update_lock'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
{"timestamp": "2022-06-17T06:21:30+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-17T06:21:32.701+00:00][7][2044][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 38 models
[2022-06-17T06:21:32.753+00:00][7][2044][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "477"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":15,\"_seq_no\":60722,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.1.2\",\"locked_at\":\"2022-06-17T06:20:24+00:00\",\"expires_at\":\"2022-06-17T06:23:26+00:00\",\"last_heartbeat_at\":\"2022-06-17T06:21:26+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Starting] Creating indices for 38 models\",\"node_name\":\"enterprise-search-bselastic-ent-75f9f87cb4-z4xsg\",\"pid\":7,\"tid\":2000}}"
  },
  "duration": 41.1,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:10:in `block in _retrieve_lock'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:10:in `block in update_lock'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
2022-06-17T06:22:19.177+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Shutting down the heartbeat timer...
[2022-06-17T06:22:19.178+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Waiting for heartbeat thread to finish...
[2022-06-17T06:22:19.273+00:00][7][2044][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-17T06:22:19+00:00\",\"expires_at\":\"2022-06-17T06:24:19+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Failed] Creating indices for 38 models: Error = Faraday::TimeoutError: Read timed out\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "177"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":25,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":60732,\"_primary_term\":1}"
  },
  "duration": 214.4,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:10:in `block in update_lock'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-17T06:22:19.280+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] The heartbeat should be down now
[2022-06-17T06:22:19.282+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Status: Releasing the lock
[2022-06-17T06:22:19.299+00:00][7][2000][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "522"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":25,\"_seq_no\":60732,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.1.2\",\"locked_at\":\"2022-06-17T06:20:24+00:00\",\"expires_at\":\"2022-06-17T06:24:19+00:00\",\"last_heartbeat_at\":\"2022-06-17T06:22:19+00:00\",\"status_update_at\":\"2022-06-17T06:20:24+00:00\",\"last_status\":\"[Failed] Creating indices for 38 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-75f9f87cb4-z4xsg\",\"pid\":7,\"tid\":2000}}"
  },
  "duration": 6.6,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:466:in `release_lock'",
    "lib/apm_helpers.class:10:in `block in release_lock'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:399:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-17T06:22:19.946+00:00][7][2000][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation?if_primary_term=1&if_seq_no=60732",
    "method": "delete",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "177"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":26,\"result\":\"deleted\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":60733,\"_primary_term\":1}"
  },
  "duration": 634.0,
  "stack": [
    "lib/db_lock.class:305:in `delete_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:470:in `release_lock'",
    "lib/apm_helpers.class:10:in `block in release_lock'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:399:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-17T06:22:19.949+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Successfully released lock
Unexpected exception while running Enterprise Search:
Error: Read timed out at

```

---

<div class="post-metadata">

**Author:** ![Artem\_Shelkovnikov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artem_shelkovnikov/32/134322_2.png) [@Artem\_Shelkovnikov](https://discuss.elastic.co/u/Artem_Shelkovnikov)\
**Post date:** [June 17, 2022, 2:50pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/5 "2022-06-17T14:50:44Z")

</div>

> [@Nids\_Rai](#):
>
> kubectl get Elasticsearch -n bselastic  
> NAME HEALTH NODES VERSION PHASE AGE  
> bselastic yellow 3 8.2.2 Ready 16d

Judging by this line, I see that Elasticsearch is not healthy - it's in Yellow status, so it's possible that part of its nodes are down or not available.

I suggest looking at Elasticsearch logs and checking the warning and error messages there - they can give you a hint what might be wrong with Elasticsearch.

I think when the cluster goes down to "green" state, the problem will go away by itself.

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 17, 2022, 5:46pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/6 "2022-06-17T17:46:45Z")

</div>

yes, But when I started it it was green , Infact now it is green .

kubectl get Elasticsearch -n bselastic

```auto
NAME HEALTH NODES VERSION PHASE AGE
bselastic green 3 8.2.2 Ready 17d

```

The the pod status -  
kubectl get pods -n bselastic

```auto
NAME READY STATUS RESTARTS AGE
[root@1175014-kubemaster01]# kubectl get pods -n bselastic
NAME READY STATUS RESTARTS AGE
bselastic-es-default-0 1/1 Running 0 17d
bselastic-es-default-1 1/1 Running 0 17d
bselastic-es-default-2 1/1 Running 0 17d
enterprise-search-bselastic-ent-7cf8754f75-dljmd 0/1 Running 0 5s
kibana-bselastic-kb-56d5f544bb-68sf5 1/1 Running 0 16d

```

Also, Is there any way I can increase the timeout in enterprise search . I feel like it would be timeout issue .

---

<div class="post-metadata">

**Author:** ![Artem\_Shelkovnikov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artem_shelkovnikov/32/134322_2.png) [@Artem\_Shelkovnikov](https://discuss.elastic.co/u/Artem_Shelkovnikov)\
**Post date:** [June 17, 2022, 6:22pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/7 "2022-06-17T18:22:06Z")

</div>

I see, so Elasticsearch looks healthy.

I noticed from the logs is that your Elasticsearch has version `8.2.2` , but Enterprise Search is `8.1.2` . Is it intended? Normally, it’s not possible to run the stack with version incompatibility, though I wonder why there is no error message indicating this. I can suggest using the same version for both Elasticsearch and Enterprise Search deployments.

Another idea that I have is that Enterprise Search instance might not have enough resources to run - I would recommend allocating at least 2GB memory for Enterprise Search environment. Can you make sure that sufficient memory is used for the deployment?

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 22, 2022, 12:09pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/8 "2022-06-22T12:09:47Z")

</div>

Hi @Artem_Shelkovnikov , I tried the same version and allocated more than 2GB of memory for the Enterprise Search environment.  
but the issue persists. Also, I noticed that: the Elasticsearch is Green and as soon as I start the App search it remains green for a while and as soon as the log shows curl\_7 error on app search,it turns to yellow.  
Do we have any way to increase the timeout of indexing in appsearch ?  
Do you have any sample setup or YAML of working app search which I can use as a reference( I have already referred the one at doc)?

still confused about this timeout .

```auto
 {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "522"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":24,\"_seq_no\":95602,\"_primary_term\":3,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-22T12:18:44+00:00\",\"expires_at\":\"2022-06-22T12:22:40+00:00\",\"last_heartbeat_at\":\"2022-06-22T12:20:40+00:00\",\"status_update_at\":\"2022-06-22T12:18:44+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-7469648db4-q9l94\",\"pid\":7,\"tid\":4004}}"
  },

```

---

<div class="post-metadata">

**Author:** ![Artem\_Shelkovnikov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artem_shelkovnikov/32/134322_2.png) [@Artem\_Shelkovnikov](https://discuss.elastic.co/u/Artem_Shelkovnikov)\
**Post date:** [June 23, 2022, 12:36pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/9 "2022-06-23T12:36:29Z")

</div>

> Do we have any way to increase the timeout of indexing in appsearch ?

I don't think it's possible, there seems to be no such setting - and the problem even happens during installation, I don't think timeout increase will help here, unfortunately.

> Also, I noticed that: the Elasticsearch is Green and as soon as I start the App search it remains green for a while and as soon as the log shows curl\_7 error on app search,it turns to yellow.

I think it's our hint here. I think that your Enterprise Search is configured correctly, but something is happening with Elasticsearch. I think inspecting the logs of Elasticsearch during the moment when you run Enterprise Search will give us required information that will help understanding what happens.

Can you share Elasticsearch logs that are displayed once you start Enterprise Search against it? It should be some [warning] or [error] line that explains why the health of Elasticsearch changes.

---

<div class="post-metadata">

**Author:** ![maryna.cherniavska](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maryna.cherniavska/32/89935_2.png) [@maryna.cherniavska](https://discuss.elastic.co/u/maryna.cherniavska)\
**Post date:** [June 23, 2022, 1:02pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/10 "2022-06-23T13:02:43Z")

</div>

> [@Nids\_Rai](#):
>
> ```auto
> {"timestamp": "2022-06-06T06:22:14+00:00", "message": "readiness probe failed", "curl_rc": "7"}
> {"timestamp": "2022-06-06T06:22:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
> {"timestamp": "2022-06-06T06:22:30+00:00", "message": "readiness probe failed", "curl_rc": "7"}
> 
> ```

Hi @Nids_Rai. These last messages aren't coming from Elasticsearch or Enterprise Search - this is `kubernetes` pod readiness probe. I don't see liveness or readiness probe explicitly configured in the yaml that you provided for either Elasticsearch or Enterprise Search. It might help if you provided that part of the configuration yaml here. I am suspecting it might be an issue of Enterprise Search just not having enough time to start before the pod is killed due to readiness probe having failed. It might be a false hypothesis, but I'd still like to see that configuration.

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 27, 2022, 7:58am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/11 "2022-06-27T07:58:32Z")

</div>

Hi @maryna.cherniavska, we have not configured any "readiness probe" explicitly. This is coming from the operator itself.

---

<div class="post-metadata">

**Author:** ![maryna.cherniavska](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/maryna.cherniavska/32/89935_2.png) [@maryna.cherniavska](https://discuss.elastic.co/u/maryna.cherniavska)\
**Post date:** [June 28, 2022, 9:30am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/12 "2022-06-28T09:30:36Z")

</div>

Hi @Nids_Rai .

Could you try to give more time to the readiness probe? It's taken from an env variable as shown [here in the docs](https://www.elastic.co/guide/en/cloud-on-k8s/current/k8s-readiness.html). So you could do this in your spec:

```auto
env:
- name: READINESS_PROBE_TIMEOUT
  value: "120"

```

And see if that gives you more time and if logs go further?

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 28, 2022, 12:32pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/13 "2022-06-28T12:32:45Z")

</div>

Hi @maryna.cherniavska  
I am using this yaml and have already configured timeout .

```auto
apiVersion: enterprisesearch.k8s.elastic.co/v1
kind: EnterpriseSearch
metadata:
  name: enterprise-search-bselastic
spec:
  version: 8.2.2
  count: 1
  elasticsearchRef:
    name: bselastic
  config:
    log_level: debug
  podTemplate:
    spec:
      nodeSelector:
        node-role.kubernetes.io/worker: worker
      containers:
      - name: enterprise-search
        env:
        - name: JAVA_OPTS
          value: -Xms2g -Xmx2g
        - name: "elasticsearch.startup_retry.interval"
          value: "30"
        - name: READINESS_PROBE_TIMEOUT
          value: "500"
        resources:
          limits:
            memory: "2Gi"
            cpu: 3
          requests:
            memory: "2Gi"
            cpu: 3

```

Even while Describe it shows that env as 500 and still getting curl\_7

```auto
[root@1175014-kubemaster01 nilesh]# kubectl describe pod enterprise-search-bselastic-ent-5668b4f75b-89m8r -n bselastic
Name: enterprise-search-bselastic-ent-5668b4f75b-89m8r
Namespace: bselastic
Priority: 0
Node: 1175018-kubeworker03.sb.rackspace.com/172.17.201.225
Start Time: Tue, 28 Jun 2022 12:30:14 +0000
Labels: common.k8s.elastic.co/type=enterprise-search
              enterprisesearch.k8s.elastic.co/name=enterprise-search-bselastic
              enterprisesearch.k8s.elastic.co/version=8.2.2
              pod-template-hash=5668b4f75b
Annotations: cni.projectcalico.org/containerID: cfcb93ef4a329c0aa05310c9a2b69682643508ac161181b87508b270fc333937
              cni.projectcalico.org/podIP: 192.168.63.209/32
              cni.projectcalico.org/podIPs: 192.168.63.209/32
              enterprisesearch.k8s.elastic.co/config-hash: 3190656219
Status: Running
IP: 192.168.63.209
IPs:
  IP: 192.168.63.209
Controlled By: ReplicaSet/enterprise-search-bselastic-ent-5668b4f75b
Containers:
  enterprise-search:
    Container ID: containerd://879f634bb1bce1e78fe1b589f18373cc5aaa0d91217baef9be1d5f321578f569
    Image: docker.elastic.co/enterprise-search/enterprise-search:8.2.2
    Image ID: docker.elastic.co/enterprise-search/enterprise-search@sha256:672d9cee9c5c80fe5540c1f0925ae271cf781f1ef237109578a258452fd948dd
    Port: 3002/TCP
    Host Port: 0/TCP
    State: Running
      Started: Tue, 28 Jun 2022 12:30:15 +0000
    Ready: False
    Restart Count: 0
    Limits:
      cpu: 3
      memory: 2Gi
    Requests:
      cpu: 3
      memory: 2Gi
    Readiness: exec [bash /mnt/elastic-internal/scripts/readiness-probe.sh] delay=60s timeout=5s period=10s #success=1 #failure=3
    Environment:
      JAVA_OPTS: -Xms2g -Xmx2g
      elasticsearch.startup_retry.interval: 30
      READINESS_PROBE_TIMEOUT: 500
      NEW_RELIC_METADATA_KUBERNETES_CLUSTER_NAME: IAD.Prod
      NEW_RELIC_METADATA_KUBERNETES_NODE_NAME: (v1:spec.nodeName)
      NEW_RELIC_METADATA_KUBERNETES_NAMESPACE_NAME: bselastic (v1:metadata.namespace)
      NEW_RELIC_METADATA_KUBERNETES_POD_NAME: enterprise-search-bselastic-ent-5668b4f75b-89m8r (v1:metadata.name)
      NEW_RELIC_METADATA_KUBERNETES_CONTAINER_NAME: enterprise-search
      NEW_RELIC_METADATA_KUBERNETES_CONTAINER_IMAGE_NAME: docker.elastic.co/enterprise-search/enterprise-search:8.2.2
      NEW_RELIC_METADATA_KUBERNETES_DEPLOYMENT_NAME: enterprise-search-bselastic-ent
    Mounts:
      /mnt/elastic-internal/es-certs from es-certs (ro)
      /mnt/elastic-internal/http-certs from elastic-internal-http-certificates (ro)
      /mnt/elastic-internal/scripts/readiness-probe.sh from readiness-probe (ro,path="readiness-probe.sh")
      /usr/share/enterprise-search/config/enterprise-search.yml from config (ro,path="enterprise-search.yml")
      /var/log/enterprise-search from logs (rw)
Conditions:
  Type Status
  Initialized True
  Ready False
  ContainersReady False
  PodScheduled True
Volumes:
  config:
    Type: Secret (a volume populated by a Secret)
    SecretName: enterprise-search-bselastic-ent-config
    Optional: false
  elastic-internal-http-certificates:
    Type: Secret (a volume populated by a Secret)
    SecretName: enterprise-search-bselastic-ent-http-certs-internal
    Optional: false
  es-certs:
    Type: Secret (a volume populated by a Secret)
    SecretName: enterprise-search-bselastic-ent-es-ca
    Optional: false
  logs:
    Type: EmptyDir (a temporary directory that shares a pod's lifetime)
    Medium:
    SizeLimit: <unset>
  readiness-probe:
    Type: Secret (a volume populated by a Secret)
    SecretName: enterprise-search-bselastic-ent-config
    Optional: false
QoS Class: Guaranteed
Node-Selectors: node-role.kubernetes.io/worker=worker
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
                 node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
  Type Reason Age From Message
  ---- ------ ---- ---- -------
  Normal Scheduled 96s default-scheduler Successfully assigned bselastic/enterprise-search-bselastic-ent-5668b4f75b-89m8r to 1175018-kubeworker03.sb.rackspace.com
  Normal Pulled 96s kubelet Container image "docker.elastic.co/enterprise-search/enterprise-search:8.2.2" already present on machine
  Normal Created 96s kubelet Created container enterprise-search
  Normal Started 95s kubelet Started container enterprise-search
  Warning Unhealthy 26s kubelet Readiness probe failed: {"timestamp": "2022-06-28T12:31:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
  Warning Unhealthy 16s kubelet Readiness probe failed: {"timestamp": "2022-06-28T12:31:34+00:00", "message": "readiness probe failed", "curl_rc": "7"}
  Warning Unhealthy 6s (x2 over 6s) kubelet Readiness probe failed: {"timestamp": "2022-06-28T12:31:44+00:00", "message": "readiness probe failed", "curl_rc": "7"}

```

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 28, 2022, 12:49pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/14 "2022-06-28T12:49:40Z")

</div>

and the latest few logs are -

```auto

[2022-06-28T12:42:15.216+00:00][8][4004][app-server][INFO]: Creating index for Actastic::Schema: workplace_search_pre_content_sources_v3
[2022-06-28T12:42:17.484+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:17.531+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":172,\"_seq_no\":171,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:11+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:11+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 43.1,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:18.030+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:17+00:00\",\"expires_at\":\"2022-06-28T12:44:17+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":173,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":172,\"_primary_term\":1}"
  },
  "duration": 493.1,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:23.034+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:23.083+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":173,\"_seq_no\":172,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:17+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:17+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 45.4,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
{"timestamp": "2022-06-28T12:42:24+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-28T12:42:24.204+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:23+00:00\",\"expires_at\":\"2022-06-28T12:44:23+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":174,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":173,\"_primary_term\":1}"
  },
  "duration": 1115.1,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:29.208+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:29.351+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":174,\"_seq_no\":173,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:23+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:23+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 138.6,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:29.530+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:29+00:00\",\"expires_at\":\"2022-06-28T12:44:29+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":175,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":174,\"_primary_term\":1}"
  },
  "duration": 172.7,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
{"timestamp": "2022-06-28T12:42:34+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-28T12:42:34.533+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:34.581+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":175,\"_seq_no\":174,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:29+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:29+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 43.7,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:35.581+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:34+00:00\",\"expires_at\":\"2022-06-28T12:44:34+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":176,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":175,\"_primary_term\":1}"
  },
  "duration": 994.0,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:40.585+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Starting] Creating indices for 41 models
[2022-06-28T12:42:40.677+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":176,\"_seq_no\":175,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:34+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:34+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 85.9,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:41.111+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:40+00:00\",\"expires_at\":\"2022-06-28T12:44:40+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":177,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":176,\"_primary_term\":1}"
  },
  "duration": 426.5,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:42.004+00:00][8][4188][app-server][DEBUG]: Summarizing metrics...
[2022-06-28T12:42:42.004+00:00][8][4188][app-server][DEBUG]: Done summarizing 0 metrics.
{"timestamp": "2022-06-28T12:42:44+00:00", "message": "readiness probe failed", "curl_rc": "7"}
[2022-06-28T12:42:45.254+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-actastic-workplace_search_pre_content_sources_v3",
    "method": "put",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"settings\":{\"index\":{\"hidden\":true,\"refresh_interval\":-1},\"number_of_shards\":1,\"auto_expand_replicas\":\"0-3\",\"priority\":250},\"mappings\":{\"dynamic\":\"strict\",\"properties\":{\"id\":{\"type\":\"keyword\"},\"created_at\":{\"type\":\"date\"},\"updated_at\":{\"type\":\"date\"},\"service_type\":{\"type\":\"keyword\"},\"frito_pie_account_id\":{\"type\":\"keyword\"},\"context\":{\"type\":\"keyword\"},\"configuration\":{\"type\":\"object\",\"enabled\":false}}},\"aliases\":{}}"
  },
  "exception": "/usr/share/enterprise-search/lib/war/lib/swiftype/es/index.class:59: Read timed out (Faraday::TimeoutError)\n",
  "duration": 30030.5,
  "stack": [
    "lib/actastic/schema.class:172:in `create_index!'",
    "lib/actastic/schema.class:195:in `create_index_and_mapping!'",
    "shared_togo/lib/shared_togo.class:912:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:910:in `block in each'",
    "shared_togo/lib/shared_togo.class:910:in `block in apply_actastic_migrations'",
    "lib/db_lock.class:182:in `with_status'",
    "shared_togo/lib/shared_togo.class:909:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo/audit_event.class:48:in `log_event_with_message'",
    "shared_togo/lib/shared_togo.class:907:in `apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:420:in `block in install!'",
    "lib/db_lock.class:171:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:45.255+00:00][8][4004][app-server][INFO]: [db_lock] [installation] Status: [Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out
[2022-06-28T12:42:45.307+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "476"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":177,\"_seq_no\":176,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:40+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:40+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Starting] Creating indices for 41 models\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 46.3,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:194:in `set_status'",
    "lib/db_lock.class:185:in `with_status'",
    "shared_togo/lib/shared_togo.class:909:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo/audit_event.class:48:in `log_event_with_message'",
    "shared_togo/lib/shared_togo.class:907:in `apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:420:in `block in install!'",
    "lib/db_lock.class:171:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.116+00:00][8][4188][app-server][DEBUG]: [db_lock] [installation] Lock heartbeat: status=[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out
[2022-06-28T12:42:46.149+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "521"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":178,\"_seq_no\":177,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:45+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:45+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 27.7,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:504:in `ensure_we_still_own_the_lock!'",
    "lib/db_lock.class:270:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:46.177+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:45+00:00\",\"expires_at\":\"2022-06-28T12:44:45+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":178,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":177,\"_primary_term\":1}"
  },
  "duration": 863.8,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:194:in `set_status'",
    "lib/db_lock.class:185:in `with_status'",
    "shared_togo/lib/shared_togo.class:909:in `block in apply_actastic_migrations'",
    "shared_togo/lib/shared_togo/audit_event.class:48:in `log_event_with_message'",
    "shared_togo/lib/shared_togo.class:907:in `apply_actastic_migrations'",
    "shared_togo/lib/shared_togo.class:420:in `block in install!'",
    "lib/db_lock.class:171:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.179+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Shutting down the heartbeat timer...
[2022-06-28T12:42:46.179+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Waiting for heartbeat thread to finish...
[2022-06-28T12:42:46.243+00:00][8][4188][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_update/installation",
    "method": "post",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": "{\"doc\":{\"last_heartbeat_at\":\"2022-06-28T12:42:46+00:00\",\"expires_at\":\"2022-06-28T12:44:46+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\"}}"
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":179,\"result\":\"updated\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":178,\"_primary_term\":1}"
  },
  "duration": 88.2,
  "stack": [
    "lib/db_lock.class:275:in `block in update_lock'",
    "config/initializers/kernel.class:5:in `retryable'",
    "lib/db_lock.class:269:in `update_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:411:in `block in setup_heartbeat_timer'"
  ]
}
[2022-06-28T12:42:46.281+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] The heartbeat should be down now
[2022-06-28T12:42:46.282+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Status: Releasing the lock
[2022-06-28T12:42:46.301+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation",
    "method": "get",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "521"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":179,\"_seq_no\":178,\"_primary_term\":1,\"found\":true,\"_source\":{\"product_version\":\"8.2.2\",\"locked_at\":\"2022-06-28T12:31:34+00:00\",\"expires_at\":\"2022-06-28T12:44:46+00:00\",\"last_heartbeat_at\":\"2022-06-28T12:42:46+00:00\",\"status_update_at\":\"2022-06-28T12:30:52+00:00\",\"last_status\":\"[Failed] Creating indices for 41 models: Error = Faraday::TimeoutError: Read timed out\",\"node_name\":\"enterprise-search-bselastic-ent-5668b4f75b-89m8r\",\"pid\":8,\"tid\":4004}}"
  },
  "duration": 13.3,
  "stack": [
    "lib/db_lock.class:299:in `_retrieve_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:466:in `release_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.389+00:00][8][4004][es][DEBUG]: {
  "request": {
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation?if_primary_term=1&if_seq_no=178",
    "method": "delete",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "176"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":180,\"result\":\"deleted\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":179,\"_primary_term\":1}"
  },
  "duration": 80.4,
  "stack": [
    "lib/db_lock.class:305:in `delete_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:470:in `release_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:413:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `new'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-06-28T12:42:46.390+00:00][8][4004][app-server][DEBUG]: [db_lock] [installation] Successfully released lock
Unexpected exception while running Enterprise Search:
Error: Read timed out at

```

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [June 28, 2022, 1:40pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/15 "2022-06-28T13:40:52Z")

</div>

Hi @Artem_Shelkovnikov and @maryna.cherniavska , Do you guys have any working YAML for enterprise search which I can refer to.

---

<div class="post-metadata">

**Author:** ![Rich\_Kuzsma](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rich_kuzsma/32/66159_2.png) [@Rich\_Kuzsma](https://discuss.elastic.co/u/Rich_Kuzsma)\
**Post date:** [June 30, 2022, 9:56pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/16 "2022-06-30T21:56:41Z")

</div>

Not sure if it helps, but I've tested Enterprise Search 8.1 using this YML file in a local single node development configuration: [Elastic Enterprise Search ECK yaml configuration files for testing · GitHub](https://gist.github.com/richkuz/76fa002cf982d2809746e686a9f1b725#file-ent-search-8-1-yml)

---

<div class="post-metadata">

**Author:** ![TWR](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TWR](https://discuss.elastic.co/u/TWR)\
**Post date:** [July 1, 2022, 12:37am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/17 "2022-07-01T00:37:31Z")

</div>

I just upgraded my on-prem implementation to 8.3 and have been seeing these Faraday errors in the logs since then. Did anyone ever discover a root cause for this?

---

<div class="post-metadata">

**Author:** ![Nids\_Rai](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nids_rai/32/103705_2.png) [@Nids\_Rai](https://discuss.elastic.co/u/Nids_Rai)\
**Post date:** [July 4, 2022, 10:47am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/18 "2022-07-04T10:47:09Z")

</div>

Hi @Rich_Kuzsma I changed my version to 8.1 as mentioned in the file but the error is same .

```auto
    "url": "https://bselastic-es-http.bselastic.svc:9200/.ent-search-db-lock-20200304/_doc/installation?if_primary_term=1&if_seq_no=85",
    "method": "delete",
    "headers": {
      "Authorization": "[FILTERED]",
      "Content-Type": "application/json",
      "x-elastic-product-origin": "enterprise-search",
      "User-Agent": "Faraday v1.8.0"
    },
    "params": null,
    "body": null
  },
  "response": {
    "status": 200,
    "headers": {
      "x-elastic-product": "Elasticsearch",
      "content-type": "application/json",
      "content-length": "174"
    },
    "body": "{\"_index\":\".ent-search-db-lock-20200304\",\"_id\":\"installation\",\"_version\":87,\"result\":\"deleted\",\"_shards\":{\"total\":2,\"successful\":2,\"failed\":0},\"_seq_no\":86,\"_primary_term\":1}"
  },
  "duration": 297.6,
  "stack": [
    "lib/db_lock.class:305:in `delete_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:470:in `release_lock'",
    "lib/apm_helpers.class:15:in `block in es_action_instrument_method'",
    "lib/db_lock.class:174:in `with_lock'",
    "shared_togo/lib/shared_togo.class:399:in `install!'",
    "config/application.class:102:in `block in Application'",
    "config/environment.class:9:in `<main>'",
    "config/environment.rb:1:in `<main>'",
    "shared_togo/lib/shared_togo/cli/command.class:37:in `initialize'",
    "shared_togo/lib/shared_togo/cli/command.class:10:in `run_and_exit'",
    "shared_togo/lib/shared_togo/cli.class:143:in `run_supported_command'",
    "shared_togo/lib/shared_togo/cli.class:125:in `run_command'",
    "shared_togo/lib/shared_togo/cli.class:112:in `run!'",
    "bin/enterprise-search-internal:15:in `<main>'"
  ]
}
[2022-07-04T10:29:54.439+00:00][7][2000][app-server][DEBUG]: [db_lock] [installation] Successfully released lock
Unexpected exception while running Enterprise Search:

```

---

<div class="post-metadata">

**Author:** ![Artem\_Shelkovnikov](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/artem_shelkovnikov/32/134322_2.png) [@Artem\_Shelkovnikov](https://discuss.elastic.co/u/Artem_Shelkovnikov)\
**Post date:** [July 4, 2022, 2:39pm UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/19 "2022-07-04T14:39:26Z")

</div>

@Nids_Rai @TWR can you share the logs of the Elasticsearch that correspond the moment when Enterprise Search starts?

At this point it looks like there is some problem with Elasticsearch configuration - might be for example connected to the lack of physical space on disk to store new indices that are generated when Enterprise Search starts.

---

<div class="post-metadata">

**Author:** ![TWR](https://avatars.discourse-cdn.com/v4/letter/t/8dc957/32.png) [@TWR](https://discuss.elastic.co/u/TWR)\
**Post date:** [July 5, 2022, 11:26am UTC](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102/20 "2022-07-05T11:26:11Z")

</div>

In my case, I have two separate 5-node clusters, each with its own two-node Enterprise Search instance, and only one of them has this problem.

Both clusters were just deployed to newly-provisioned 5 VMs, so each cluster has a full terabyte of storage available.

Before I dive into the logs, would it be worthwhile to disable this setting in enterprise-search.yml:

`allow_es_settings_modification`

Enterprise Search needs one-time permission to alter Elasticsearch settings. Ensure the Elasticsearch settings are correct, then set the following to true. Or, adjust Elasticsearch’s config/elasticsearch.yml instead.

`allow_es_settings_modification: false`

The documentation [Configuration | Elastic Enterprise Search Documentation [8.3] | Elastic](https://www.elastic.co/guide/en/enterprise-search/current/configuration.html#configuration-file) says to set this to true OR adjust Elasticsearch's config instead, but it doesn't indicate what we would do in the elasticearch.yml instead...

[Next page](https://discuss.elastic.co/t/unexpected-exception-while-running-enterprise-search-error-read-timed-out-at/307102.md?page=2)