Elasticsearch 2.3 as primary data store?

Elastic Stack Elasticsearch
1

I see several threads in the forums where others are stating ES will lose data, but a lot of the posts are from 2013. Anyone recommend ES 2.3 as the primary data store at this iteration of ES?

For my use case we do not need transactional data rollbacks, etc. that you would find in Couchbase or Mongo. Also, we would make use of the close/open index endpoint to effectively TTL cold data, but be able to "open" retrieve it if needed.

It seems like as long as the proper amount of replica's are being set, and since we're hosting on ES Cloud we have daily backups it would be as reliable/resilient as as couchbase for my use case.

Would be very nice to not have to store the same data in couchbase that is stored in _source already in ES. Also simplifies application requirements of course.

Thoughts?

1 Like
2

If you are on Cloud, you have snapshots every 30 minutes.

1 Like
3

Hi Zelfapp,

Some key points first:

  • We do not advertise Elasticsearch as a primary data store but many customers are using it as a primary data store.
  • Elasticsearch is an eventually consistent, near-realtime storage engine.
  • Elasticsearch is not ACID compliant like a database.
  • Elastic Cloud provides snapshots every 30 minutes.

That being said, the current status of Elasticsearch resiliency is documented here:
https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html

There will be more resiliency improvements coming in v5.x release but many improvements have been realised in v2.x and back ported to v1.x.

This decision is risk analysis around how much data you could lose and your ability to replay/restore that data to a point in time if needed or ability to cope with some data missing if you choose not to restore. It depends on your particular use case and tolerance for data loss. For some things it's critical and for other things it might not matter so much for some data to be missing or slightly out of date.

With the ability to perform incremental snapshots regularly, you should be able to restore to a fairly recent point in time to get data back. There's also many other techniques to limit loss or downtime such as how many nodes you have, how many replicas, your disk raid configuration,etc.

Also the use of brokers/queues such as Kafka/RabbitMQ/others can help guard against data loss by keeping recent updates in a separate queue for the ability to play back data from a point in time. Brokers also afford you other benefits for cluster maintenance without data loss by queuing updates during maintenance for flowing into the cluster after maintenance is complete.