Do we need to avoid using Elastic Search as primary DB?

Elastic Stack Elasticsearch
1

We are using Elasticsearch as primary db which includes entire ELK, unfortunately last week we lost huge data from Elasticsearch during server restart.
1) Do we need to avoid using Elastic Search as primary DB? If Yes, can you suggest appropriate Primary DB to use before Elasticsearch?
2) How do i handshake with elasticsearch from any primary db(Suggested by you or mysql or snowflakes etc)?

2

That indicates a serious issue in your infrastructure or orchestration process and not really anything to do with using Elasticsearch (assuming you are using a reasonably recent version anyway). The things that could lead to data loss during an Elasticsearch restart would also lead to data loss when using any other database too.

The reference manual contains extensive guidance on configuring your cluster for resilience.

3 Likes
3

Welcome to our community! :smiley:

To expand on David's comments, if you shared more about what happened we might be able to resolve your issue.

4

Thanks for your response and reference..I'm using docker container for Elasticsearch. Below is the docker-compose yaml script where i lost data..Please help to confirm that i have necessary parameters..

elasticsearch:
    container_name: nids-elk-es-cntr
    image: elasticsearch:7.9.1
    environment:
      - cluster.name=nids-cluster
      - bootstrap.memory_lock=true
      - discovery.type=single-node
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
      - ELASTIC_USERNAME=elastic
      - ELASTIC_PASSWORD=$ELASTIC_PASSWORD
      - xpack.security.enabled=$ELASTIC_SECURITY
    volumes:
      - nids_data:/usr/share/elasticsearch/data:rw
    ulimits:
      memlock:
        soft: -1
        hard: -1
    ports:
      - 9200:9200
      - 9300:9300
    depends_on:
      - kafka
    stdin_open: true
    tty: true
    restart: always
    networks:
      - nids
    logging:
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "50"
5

Elasticsearch 7.9 is long past EOL. It doesn't have any known data-loss bugs, but still it's pretty old and has other known issues. You should upgrade to a supported version as a matter of urgency.

A single-node cluster will not give you high availability.

The most common mistake that leads to data loss on a restart is putting all of your data somewhere ephemeral. Are you sure that (a) this volume persists across restarts and (b) Elasticsearch is actually putting its data there?

1 Like
6

Elasticsearch 7.9 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns :elasticheart: )

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.