# Initializing elasticsearch db

**URL:** <https://discuss.elastic.co/t/initializing-elasticsearch-db/227391>\
**Category:** Elasticsearch\
**Created:** [April 9, 2020, 6:43pm UTC](https://discuss.elastic.co/t/initializing-elasticsearch-db/227391 "2020-04-09T18:43:26Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![Paul\_Stevens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_stevens/32/48977_2.png) [@Paul\_Stevens](https://discuss.elastic.co/u/Paul_Stevens)\
**Post date:** [April 9, 2020, 6:43pm UTC](https://discuss.elastic.co/t/initializing-elasticsearch-db/227391/1 "2020-04-09T18:43:27Z")

</div>

Openstack running VM(s) running 'Red Hat Enterprise Linux Server release 7.7 (Maipo)'  
elasticsearch-oss:6.8.2  
Kibana was the first clue things were bad.

This is lab. I would be happy to initialize database and loose all data if that was possible.  
Any such initialize options?

Here is the history/issue:

My master and data nodes are no longer starting. I had a power outage, which resulted in fsck error. I ran successfully fsck manually on a /dev/vdX.

# Now Current errors(note i have other masters)

```auto
elastic-elasticsearch-client-6458757cf5-x4m47 0/1 Running 0
elastic-elasticsearch-data-0 1/1 Running 0
elastic-elasticsearch-data-1 0/1 CrashLoopBackOff 6
elastic-elasticsearch-master-0 0/1 CrashLoopBackOff 266 
elastic-exporter-elasticsearch-exporter-6c8c8c8854-n2f4r 1/1 Running 0
logs-fluentd-elasticsearch-4g5n4 1/1 Running 0
logs-fluentd-elasticsearch-8hdtt 1/1 Running            

    **DATA-1 NODE EVENTS:**  
    Conditions:
      Type Status
      Initialized True 
      Ready False 
      ContainersReady False 
      PodScheduled True 
    Volumes:
      data:
        Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
        ClaimName: data-occne-elastic-elasticsearch-data-1
        ReadOnly: false
      config:
        Type: ConfigMap (a volume populated by a ConfigMap)
        Name: occne-elastic-elasticsearch
        Optional: false
      occne-elastic-elasticsearch-data-token-bjlht:
        Type: Secret (a volume populated by a Secret)
        SecretName: occne-elastic-elasticsearch-data-token-bjlht
        Optional: false
    QoS Class: Burstable
    Node-Selectors: <none>
    Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                     node.kubernetes.io/unreachable:NoExecute for 300s

    Events:
      Type Reason Age From Message
      ---- ------ ---- ---- -------
      Warning FailedMount 47m (x1272 over 44h) kubelet, vcne-k8s-node-4 MountVolume.MountDevice failed for volume "pvc-ff0ede3e-d3a0-4476-87a2-2ecd6ef4fe2f" : 'fsck' found errors on device /dev/disk/by-id/virtio-9242dfe6-ee8d-42b4-9 but could not correct them: fsck from util-linux 2.23.2
    /dev/vdb contains a file system with errors, check forced.
    /dev/vdb: Directory inode 1573992, block #0, offset 0: directory corrupted

    /dev/vdb: UNEXPECTED INCONSISTENCY; RUN fsck MANUALLY.
      (i.e., without -a or -p options)
    .
      Warning FailedMount 37m (x1154 over 44h) kubelet, vcne-k8s-node-4 Unable to mount volumes for pod "occne-elastic-elasticsearch-data-1_occne-infra(501dcb98-cd72-44b6-9ffc-e553d65ad98a)": timeout expired waiting for volumes to attach or mount for pod "occne-infra"/"occne-elastic-elasticsearch-data-1". list of unmounted volumes=[data]. list of unattached volumes=[data config occne-elastic-elasticsearch-data-token-bjlht]
      Warning BackOff 3m20s (x152 over 37m) kubelet, vcne-k8s-node-4 Back-off restarting failed container

    **MASTER EVENTS:**
        Conditions:
          Type Status
          Initialized True 
          Ready False 
          ContainersReady False 
          PodScheduled True 
        Volumes:
          data:
            Type: PersistentVolumeClaim (a reference to a PersistentVolumeClaim in the same namespace)
            ClaimName: data-occne-elastic-elasticsearch-master-0
            ReadOnly: false
          config:
            Type: ConfigMap (a volume populated by a ConfigMap)
            Name: occne-elastic-elasticsearch
            Optional: false
          occne-elastic-elasticsearch-master-token-5twdw:
            Type: Secret (a volume populated by a Secret)
            SecretName: occne-elastic-elasticsearch-master-token-5twdw
            Optional: false
        QoS Class: Burstable
        Node-Selectors: <none>
        Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
                         node.kubernetes.io/unreachable:NoExecute for 300s
        Events:
          Type Reason Age From Message
          ---- ------ ---- ---- -------
          Warning BackOff <invalid> (x6440 over 23h) kubelet, vcne-k8s-node-1 Back-off restarting failed container

```

Any help appreciated.  
Paul

---

<div class="post-metadata">

**Author:** ![oneoneonepig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oneoneonepig/32/66018_2.png) [@oneoneonepig](https://discuss.elastic.co/u/oneoneonepig)\
**Post date:** [April 10, 2020, 3:07am UTC](https://discuss.elastic.co/t/initializing-elasticsearch-db/227391/2 "2020-04-10T03:07:55Z")

</div>

@Paul_Stevens after you perform fsck, could you try to restart the pod manually?

---

<div class="post-metadata">

**Author:** ![Paul\_Stevens](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/paul_stevens/32/48977_2.png) [@Paul\_Stevens](https://discuss.elastic.co/u/Paul_Stevens)\
**Post date:** [April 10, 2020, 4:19am UTC](https://discuss.elastic.co/t/initializing-elasticsearch-db/227391/3 "2020-04-10T04:19:04Z")

</div>

I did restart pods manually. Somehow the sequence of doing fsck(s) and restarting master-0 I had all 3 master running. Then reset data-X pods and now i am running again. Learning the process....  
Thanks for the update we can close this.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 8, 2020, 4:19am UTC](https://discuss.elastic.co/t/initializing-elasticsearch-db/227391/4 "2020-05-08T04:19:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
