# Elasticsearch as a atabase **URL:** \ **Category:** Elasticsearch\ **Created:** [December 5, 2012, 12:03pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952 "2012-12-05T12:03:44Z")\ **Posts on this page:** 11\ **Page:** 1
**Author:** ![Maciej\_Dziardziel](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@Maciej\_Dziardziel](https://discuss.elastic.co/u/Maciej_Dziardziel)\ **Post date:** [December 5, 2012, 12:03pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/1 "2012-12-05T12:03:44Z")
Hi I am considering using ES as a key-value storage (as a primary data store, not cache). Is it suitable for this purpose? Initially it looks really interesting - i like easy way of adding nodes, thrift, realtime kv api and other features, however there are few topics where documentation is not clear for me, so i have few questions: 1. What happens when a new document is indexed? Can i be sure that it will be stored on the disk even if a node would go down a millisecond later? Documentation mentions hat index only succeeds if "a quorum (\>replicas/2+1) of active shards are available" - does it mean that it actually stores data on those nodes, or that they only need to be "not known to be down at the moment of index"? 2. How to perform a backup and restore of whole index? 3. How reliable would ES be when used for this purpose? Thanks for any help -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 5, 2012, 12:28pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/2 "2012-12-05T12:28:25Z")
Hello, On Wed, Dec 5, 2012 at 2:03 PM, Maciej Dziardziel [fiedzia@gmail.com](mailto:fiedzia@gmail.com) wrote: > Hi > > I am considering using ES as a key-value storage (as a primary data store, > not cache). Is it suitable for this purpose? I think it is, for most use cases. Of course, it depends on what features you need and whether ES can provide them. > Initially it looks really interesting - i like easy way of adding nodes, > thrift, realtime kv api and other features, however there are few topics > where documentation is not clear for me, > so i have few questions: > > 1. What happens when a new document is indexed? Can i be sure that it will > be stored on the disk even if a node would go down a millisecond later? Yes. You'll get an OK for your indexing operation once it has been stored in your transaction log, which is flushed to your index at times (according to your settings): > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... So even if the doc doesn't have a chance to get into your index, it can get there after restarting the node. > Documentation mentions hat index only succeeds if "a quorum > (\>replicas/2+1) of active shards are available" - does it mean that it > actually stores data > on those nodes, Yes. > or that they only need to be "not known to be down at the moment of index"? > > 1. How to perform a backup and restore of whole index? You need to disable flush using the Indices Update Settings API: > **[Elastic — The Search AI Company](https://www.elastic.co)** > > Power insights and outcomes with The Elastic Search AI Platform. See into your data and find answers that matter with enterprise solutions designed to help you accelerate time to insight. Try Elastic ... Then copy what's in your data directory, then enable flush again. To restore, you'd have to stop the same ES node, restore what you backed up and restart the node. That might get tricky on a multiple-node cluster, because you have to make sure that the data you restore doesn't conflict with what is already in the cluster metadata. If you want some code examples to begin with, here's something I used for daily indices containing logs: > > 1. How reliable would ES be when used for this purpose? You can use replicas to make sure your cluster survives even if some individual nodes go down. The nice part about this is that you can change the number of replicas on a live cluster. And, searches run on replicas as well. So if you add some more nodes and replicas, your system should handle more concurrent queries, besides being more fault-tolerant. If you need extra reliability, I think you should have a safety net of either backing up your indices regularly, or keeping the source in another store as well, so you can reindex it. ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![Andrej](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Andrej](https://discuss.elastic.co/u/Andrej)\ **Post date:** [December 5, 2012, 12:55pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/3 "2012-12-05T12:55:06Z")
Hi, As Radu pointed out already there are ways to achieve what you want, actually we came up with an equal idea a while ago and used it successfully (storing several terrabytes of data). Elasticsearch provides a lot of features to make your store reliable, so I wouldnt worry here (actually we never lost data in years due to problems in elasticsearch). Nevertheless, to share our experience, we moved away from this, because when you just want a keyvalue-store elasticsearch is, IMHO, a bit an overkill. You ask what happens when a document is indexed, but do you want to index documents or just store them? Indexing, with all steps included, takes time, just storing a document is much faster ( we tried MongoDB, inserting speed is up to 10 times faster). Also you have an overhead due to the index structure (look at all the lucene files) when considering file size, might turn out to be a problem when you are limited e.g. on ssd drives. One point considering fetching would be to look at a routing strategy, so you dont have to talk to the whole index but just to one shard, it for sure improves performance. When your scenario fits to it you can make it working in a reliable way, no doubt here, but if you have special constraints on performance or number of documents I would think about what is actually information you want to search on and what to just store, it might be more effective to separate it. Stores are good for storing, search solutions for searching 😉 Greets Andrej -- ---
**Author:** ![Maciej\_Dziardziel](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@Maciej\_Dziardziel](https://discuss.elastic.co/u/Maciej_Dziardziel)\ **Post date:** [December 5, 2012, 1:00pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/4 "2012-12-05T13:00:00Z")
On Wednesday, December 5, 2012 12:28:25 PM UTC, Radu Gheorghe wrote: > Hello, > > > Initially it looks really interesting - i like easy way of adding nodes, > > thrift, realtime kv api and other features, however there are few topics > > where documentation is not clear for me, > > so i have few questions: > > > > 1. What happens when a new document is indexed? Can i be sure that it > > will be stored on the disk even if a node would go down a millisecond later? > > Yes. You'll get an OK for your indexing operation once it has been stored > in your transaction log, which is flushed to your index at times (according > to your settings): > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/translog.html) That's clear now - thanks. > 1. How to perform a backup and restore of whole index? > > > > > You need to disable flush using the Indices Update Settings API: > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html) > > Then copy what's in your data directory, then enable flush again. But this only copy data from single node. My intention is to split it between several nodes. Is there any other way, or do i need to backup all nodes separately? > 1. How reliable would ES be when used for this purpose? You can use replicas to make sure your cluster survives even if some individual nodes go down. The nice part about this is that you can change the number of replicas on a live cluster. And, searches run on replicas as well. So if you add some more nodes and replicas, your system should handle more concurrent queries, besides being more fault-tolerant. If you need extra reliability, I think you should have a safety net of either backing up your indices regularly, or keeping the source in another store as well, so you can reindex it. My plan is to use it as the only storage do regular backups, ## Thanks for your help. [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene > -- ---
**Author:** ![Maciej\_Dziardziel](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@Maciej\_Dziardziel](https://discuss.elastic.co/u/Maciej_Dziardziel)\ **Post date:** [December 5, 2012, 1:19pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/5 "2012-12-05T13:19:42Z")
On Wednesday, December 5, 2012 12:55:06 PM UTC, Andrej Rosenheinrich wrote: > Hi, > > As Radu pointed out already there are ways to achieve what you want, > actually we came up with an equal idea a while ago and used it successfully > (storing several terrabytes of data). Elasticsearch provides a lot of > features to make your store reliable, so I wouldnt worry here (actually we > never lost data in years due to problems in elasticsearch). Nevertheless, > to share our experience, we moved away from this, because when you just > want a keyvalue-store elasticsearch is, IMHO, a bit an overkill. You ask > what happens when a document is indexed, but do you want to index documents > or just store them? Just store, as pure k/v storage. > Indexing, with all steps included, takes time, just storing a document is > much faster ( we tried MongoDB, inserting speed is up to 10 times faster). > Also you have an overhead due to the index structure (look at all the > lucene files) when considering file size, might turn out to be a problem > when you are limited e.g. on ssd drives. Can you elaborate on that? I am not familiar with lucene. What overhead can i expect? How it compares with mongo in your case? > One point considering fetching would be to look at a routing strategy, so > you dont have to talk to the whole index but just to one shard, it for sure > improves performance. Is there anything i need to do? Documentation mentions that there is a default routing based on hash of \_id field, so it seems i don't need to worry about that. > When your scenario fits to it you can make it working in a reliable way, > no doubt here, but if you have special constraints on performance or number > of documents I would think about what is actually information you want to > search on and what to just store, it might be more effective to separate > it. Stores are good for storing, search solutions for searching 😉 Yes, i keep hearing that. However i am going to use ES for searching, and if could use it for k/v as well, that would greatly simplified my environment - unless of course ES is completely unsuitable for this purpose, or there is something better. For now i'd choose it over mongo for easier maintenance, and over riak for reach set of operations. I'll run benchmarks to be sure it does what i need, but it seems to meet my requirements. -- ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 5, 2012, 1:35pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/6 "2012-12-05T13:35:31Z")
Hello, On Wed, Dec 5, 2012 at 3:00 PM, Maciej Dziardziel [fiedzia@gmail.com](mailto:fiedzia@gmail.com) wrote: > On Wednesday, December 5, 2012 12:28:25 PM UTC, Radu Gheorghe wrote: > > > Hello, > > > > > Initially it looks really interesting - i like easy way of adding nodes, > > > thrift, realtime kv api and other features, however there are few topics > > > where documentation is not clear for me, > > > so i have few questions: > > > > > > 1. What happens when a new document is indexed? Can i be sure that it > > > will be stored on the disk even if a node would go down a millisecond later? > > > > Yes. You'll get an OK for your indexing operation once it has been stored > > in your transaction log, which is flushed to your index at times (according > > to your settings): > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/**guide/reference/index-modules/)\*\* > > translog.html[http://www.elasticsearch.org/guide/reference/index-modules/translog.html](http://www.elasticsearch.org/guide/reference/index-modules/translog.html) > > That's clear now - thanks. > > > 1. How to perform a backup and restore of whole index? > > > > > > > > > You need to disable flush using the Indices Update Settings API: > > [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/**guide/reference/api/admin-)\*\* > > indices-update-settings.html[http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html](http://www.elasticsearch.org/guide/reference/api/admin-indices-update-settings.html) > > > > Then copy what's in your data directory, then enable flush again. > > But this only copy data from single node. My intention is to split it > between several nodes. Is there any other way, > or do i need to backup all nodes separately? You can either backup all your nodes separately, or you can do a scroll[0] through large portions of your data (maybe complete indices) and get the source stored or indexed somewhere else as well, so you can reindex it back if you need to. Backing up your data directory will back up your whole index (data + index), as opposed to only your initial data. [0] [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/api/search/scroll.html) ## Best regards, Radu [http://sematext.com/](http://sematext.com/) -- Elasticsearch -- Solr -- Lucene -- ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 5, 2012, 6:32pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/7 "2012-12-05T18:32:58Z")
I'm not familiar with your key/value data structure, there is so much that depends on the data. But here are some general remarks. If you only store data and never index and never search with queries, you don't really use Elasticsearch to its full extent. Elasticsearch is built upon Lucene, which is an inverted index, not a key/value store. Right now, each document will get indexed / stored again if just a single field value in a document changes, and a single value change triggers a write to the whole Lucene index files. This is a large I/O overhead compared to writes into a key/value store. With Lucene 4, Elasticsearch will be able to offer DocValues, which will improve the situation a lot. Note, MongoDB offers update-in-place, at the downside it comes with massive space overhead. Be also aware of the features you may want in Elasticsearch, for example storing blobs, if the values are large and opaque in your key/value store. Elasticsearch speaks JSON over HTTP REST, so you have to serialize each blob to plain text. There is no method yet of storing binary objects unencoded, only by using base64 (and optionally compressed). Jörg -- ---
**Author:** ![mvg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mvg/32/98890_2.png) [@mvg](https://discuss.elastic.co/u/mvg)\ **Post date:** [December 6, 2012, 5:32pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/8 "2012-12-06T17:32:36Z")
If you choose to use ES only as a key / value store, you can consider to set `index` to `no` in your mappings. This way the fields in your json document source don't get indexed into the inverted index. This lowers the mentioned I/O overhead. On 5 December 2012 19:32, Jörg Prante [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > I'm not familiar with your key/value data structure, there is so much that > depends on the data. But here are some general remarks. > > If you only store data and never index and never search with queries, you > don't really use Elasticsearch to its full extent. > > Elasticsearch is built upon Lucene, which is an inverted index, not a > key/value store. Right now, each document will get indexed / stored again if > just a single field value in a document changes, and a single value change > triggers a write to the whole Lucene index files. This is a large I/O > overhead compared to writes into a key/value store. With Lucene 4, > Elasticsearch will be able to offer DocValues, which will improve the > situation a lot. Note, MongoDB offers update-in-place, at the downside it > comes with massive space overhead. > > Be also aware of the features you may want in Elasticsearch, for example > storing blobs, if the values are large and opaque in your key/value store. > Elasticsearch speaks JSON over HTTP REST, so you have to serialize each blob > to plain text. There is no method yet of storing binary objects unencoded, > only by using base64 (and optionally compressed). > > Jörg > > -- -- Met vriendelijke groet, Martijn van Groningen -- ---
**Author:** ![Maciej\_Dziardziel](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@Maciej\_Dziardziel](https://discuss.elastic.co/u/Maciej_Dziardziel)\ **Post date:** [December 7, 2012, 9:48am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/9 "2012-12-07T09:48:45Z")
On Wednesday, December 5, 2012 6:32:58 PM UTC, Jörg Prante wrote: > Be also aware of the features you may want in Elasticsearch, for example > storing blobs, if the values are large and opaque in your key/value > store. Elasticsearch speaks JSON over HTTP REST, so you have to serialize > each blob to plain text. There is no method yet of storing binary objects > unencoded, only by using base64 (and optionally compressed). That's ok for me, i'll only store text anyway. As for protocol serialization, ES supports thrift, so this is not a problem. > Jörg -- ---
**Author:** ![Maciej\_Dziardziel](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@Maciej\_Dziardziel](https://discuss.elastic.co/u/Maciej_Dziardziel)\ **Post date:** [December 7, 2012, 9:49am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/10 "2012-12-07T09:49:40Z")
On Thursday, December 6, 2012 5:32:36 PM UTC, Martijn v Groningen wrote: > If you choose to use ES only as a key / value store, you can consider > to set `index` to `no` in your mappings. This way the fields in your > json document source don't get indexed into the inverted index. This > lowers the mentioned I/O overhead. > > That makes sense,thanks. -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:00am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-atabase/9952/11 "2017-07-06T03:00:52Z")