Elasticsearch as a primary storage

[xtapodi]

We have lots of users(30-200 million). Each user has N(30-100) attributes. The attribute can be of type integer, text, timestamp or counter. Schema is not defined.

We are looking for a database to which we can index and retrieve(GET API) users in the most efficient way(low response time). We also have the requirement to search for users but this is out of this topic. We also need to have strong consistent and high available data.

I already know that GET API provides consistent data, but I'm not sure if Elasticsearch is the right choice for this case. We have tried Cassandra but the read performance was low(high disk IOPS).

[DineshNaik (Dinesh Kumar Naik)]

Please read this

Elasticsearch as a primary database

No. Not completely. Some shards might be missing. Not all. It depends but it might be hard to recover from that situation. Depending on how the RED actually occurred. The one I already pasted: Some customers use Elasticsearch as a primary datastore, some set-up comprehensive back-up solutions using features such as our Snapshot and Restore, while others use Elasticsearch in conjunction with a data storage system like Hadoop or even flat files. Elasticsearch can be used for so many differe…

You will get an idea!

[xtapodi]

Thanks DineshNaik. I have already read this answer. My question is more about performance.

[stephenb (Stephen Brown)]

Hi @xtapodi Welcome to the community and Thanks for considering Elasticsearch.

Elasticsearch when configured correctly (hardware software and index strategy) is highly performant We would need to learn a little bit about what you're trying to do.

even 200 million records with the 100 attributes each is still a relatively small data set with respect to Elasticsearch.

I know you said no schema but there will be a schema whether you set it or not and then with proper queries you should be able to query across that entire data set with highly performant low latency queries.

I would suggest setting up the a node or a cluster or better yet just try a Elastic Cloud and do some testing.

[warkolm (Mark Walkom)]

@DineshNaik

Please read this

That's nearly 5 years old. I'd encourage you to read up on resilience changes that have been made since then.

[dadoonet (David Pilato)]

Ha! Right. I wrote this answer a looong time ago and all the great work about sequence numbers and resilience have been done since then.

IMO the source of truth (which is always up to date) is Elasticsearch Resiliency Status | Elastic