I have recently started exploring Elastic Stack and came across your Beginner's series which has solved many doubts about Elasticsearch but I am still not sure why we not use Database and prefer Elasticsearch, and why/how we store data in elasticsearch if its in our database. I know elasticsearch is a search engine which can help in finding relevant results. So, is there any other reason to use elasticsearch or is it fine? It'd be really helpful if you can solve this doubt.
If your data structure is somewhat flat you will get super fast result then normal RDBMS database.
storing data in elk is easy, retrive is easy, delete is easy if you have uniq key. update is costly operation.
you can do all these operation using python, node.js, rest api and many other way.
The easy part comes from the fact that you can do everything through HTTP(S) calls using a well defined API.
The unique key is just the record or document identifier. In Elasticsearch it is the "_id" field. In RDBMS's it is often named "id" or something similar.
It's not using it over a RDBMS, it's depends on your use case... as elasticforme said elastic have merit compare to RDBMS as it's fast when you have flat data structure and easy to use, as bshaw also said.
It's really depends on your use case, I use elastic in several web application so maybe similar to your use case. Depends on the model I store in DB, both or only elastic.... I even store data in flat file.
Here how I decide where to store my data, maybe other people have different usage but I hope it can help you get better insight.
About data that I store only in DB:
it's when there's few data with few access for example settings for one specific option for customer and I know that I will never have more than 1K records. <--- why I store in database here it's because of elastic the more index you have the more you need memory for your server, storing in DB cost less.
data that need to be update often like counter <-- this one also depend on your need I also use what I call the "log way storage" saving data in periodical index (daily, weekly, monthly... index) and instead of updating one field I aggregate on a period.
About data I store in both (DB and elastic):
it's about the CQRS pattern I found an old link that explain about:
I use it for data that need frequent and fast access on read, I can handle peak traffic without high latency or crashing the DB with limit connection, Elastic is really shining here.
When I need to make statistics dashboard I store my business data as usual in DB and on save I flatten the data to also store in elastic, here is maybe the more complicated parts as you need to know your data to apply the correct mapping, for which statistics you want to provide. And having your original data in DB provide you one backup as you can reindex your data using build-in pipeline or home made script to transform your data in something that fit your changing needs. It give you a great flexibility and here again Elastic is really good to get data extremely fast for your statistics.
And last, only elastic:
When I have flat data like settings that have a one to one relation that can be retrieved by it id, like a Redis cache but with data persistence.
I hope it reply to your question. Sorry for "the depends on your case" I tried to give the more details example in my reply but it really depends on your case and data.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
