# ElasticSearch as a database **URL:** \ **Category:** Elasticsearch\ **Created:** [May 16, 2011, 4:06pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414 "2011-05-16T16:06:21Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![Ville](https://avatars.discourse-cdn.com/v4/letter/v/dec6dc/32.png) [@Ville](https://discuss.elastic.co/u/Ville)\ **Post date:** [May 16, 2011, 4:06pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/1 "2011-05-16T16:06:21Z")
I'm planning to use ES as a storage for JSON documents. The idea is to reference each document with index/type/id and perform search queries when needed. I've heard some concerns that Lucene based indexes are not reliable and that some documents might be missed. Is this still a valid concern? E.g. if I were to store items like this {..."done":"true"/"false"...}, is it possible that when searching for documents where done=false some documents are never received by the client? ---
**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\ **Post date:** [May 16, 2011, 5:08pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/2 "2011-05-16T17:08:46Z")
Lucene is reliable we never ever had those problems. Neither in pure lucene, solr or elasticsearch. That said it is easily possible to 'overwrite' an existing document, but with the versioning feature of Elasticsearch you can prevent this. PS: I'm actually using ES as my NoSql store for [jetwick.com](http://jetwick.com) 😉 On May 16, 6:06 pm, Ville [villematti.toivo...@gmail.com](mailto:villematti.toivo...@gmail.com) wrote: > I'm planning to use ES as a storage for JSON documents. The idea is to > reference each document with index/type/id and perform search queries > when needed. > > I've heard some concerns that Lucene based indexes are not reliable > and that some documents might be missed. Is this still a valid > concern? E.g. if I were to store items like this > {..."done":"true"/"false"...}, is it possible that when searching for > documents where done=false some documents are never received by the > client? ---
**Author:** ![timscott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timscott/32/3220_2.png) [@timscott](https://discuss.elastic.co/u/timscott)\ **Post date:** [May 16, 2011, 7:31pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/3 "2011-05-16T19:31:09Z")
Your example of retrieving documents based on a boolean is supported in ES by filters. My app depends on using filters to narrow searches, and I have not had any trouble (although I'm only in alpha). If you only ever want filter (and not search indexed data), you'd probably be better off with one of the many other NoSql alternatives, IMO. On May 16, 11:06 am, Ville [villematti.toivo...@gmail.com](mailto:villematti.toivo...@gmail.com) wrote: > I'm planning to use ES as a storage for JSON documents. The idea is to > reference each document with index/type/id and perform search queries > when needed. > > I've heard some concerns that Lucene based indexes are not reliable > and that some documents might be missed. Is this still a valid > concern? E.g. if I were to store items like this > {..."done":"true"/"false"...}, is it possible that when searching for > documents where done=false some documents are never received by the > client? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 16, 2011, 7:51pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/4 "2011-05-16T19:51:30Z")
Only filtering on boolean queries is supported by constant\_score, or you can also use things like term _query_ to query it, not necessarily a filter. On Monday, May 16, 2011 at 10:31 PM, Tim Scott wrote: Your example of retrieving documents based on a boolean is supported > in ES by filters. My app depends on using filters to narrow searches, > and I have not had any trouble (although I'm only in alpha). > > If you only ever want filter (and not search indexed data), you'd > probably be better off with one of the many other NoSql alternatives, > IMO. > > On May 16, 11:06 am, Ville [villematti.toivo...@gmail.com](mailto:villematti.toivo...@gmail.com) wrote: > > > I'm planning to use ES as a storage for JSON documents. The idea is to > > reference each document with index/type/id and perform search queries > > when needed. > > > > I've heard some concerns that Lucene based indexes are not reliable > > and that some documents might be missed. Is this still a valid > > concern? E.g. if I were to store items like this > > {..."done":"true"/"false"...}, is it possible that when searching for > > documents where done=false some documents are never received by the > > client? ---
**Author:** ![Ville](https://avatars.discourse-cdn.com/v4/letter/v/dec6dc/32.png) [@Ville](https://discuss.elastic.co/u/Ville)\ **Post date:** [May 16, 2011, 9:25pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/5 "2011-05-16T21:25:55Z")
Thank you for the quick replies. I'm going to do a lot of other kind of queries also so I would like to use ES. However, the main point in my original post was the reliability of Lucene/ES itself. My collegues have expressed concerns that Lucene does not always "find" what it should. To me this would seem like a bug but I couldn't find any bug reports with google related to these concerns. Based on your responses above, it seems like that you consider ES reliable in terms of correctness. My concern could be summarized as "Is ES less reliable for indexing than e.g. native MySQL indexes?". I guess not? On 16 touko, 22:51, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > Only filtering on boolean queries is supported by constant\_score, or you can also use things like term _query_ to query it, not necessarily a filter.On Monday, May 16, 2011 at 10:31 PM, Tim Scott wrote: > > Your example of retrieving documents based on a boolean is supported > > > in ES by filters. My app depends on using filters to narrow searches, > > and I have not had any trouble (although I'm only in alpha). > > > If you only ever want filter (and not search indexed data), you'd > > probably be better off with one of the many other NoSql alternatives, > > IMO. > > > On May 16, 11:06 am, Ville [villematti.toivo...@gmail.com](mailto:villematti.toivo...@gmail.com) wrote: > > > > > I'm planning to use ES as a storage for JSON documents. The idea is to > > > reference each document with index/type/id and perform search queries > > > when needed. > > > > I've heard some concerns that Lucene based indexes are not reliable > > > and that some documents might be missed. Is this still a valid > > > concern? E.g. if I were to store items like this > > > {..."done":"true"/"false"...}, is it possible that when searching for > > > documents where done=false some documents are never received by the > > > client? ---
**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\ **Post date:** [May 16, 2011, 9:28pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/6 "2011-05-16T21:28:14Z")
I am not familiar with any bugs in Lucene search. It might be that your colleague got confused as to when and how to refresh Lucene in order to make documents visible for searching. On Tuesday, May 17, 2011 at 12:25 AM, Ville wrote: > Thank you for the quick replies. I'm going to do a lot of other kind > of queries also so I would like to use ES. > > However, the main point in my original post was the reliability of > Lucene/ES itself. My collegues have expressed concerns that Lucene > does not always "find" what it should. To me this would seem like a > bug but I couldn't find any bug reports with google related to these > concerns. > > Based on your responses above, it seems like that you consider ES > reliable in terms of correctness. My concern could be summarized as > "Is ES less reliable for indexing than e.g. native MySQL indexes?". I > guess not? > > On 16 touko, 22:51, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Only filtering on boolean queries is supported by constant\_score, or you can also use things like term _query_ to query it, not necessarily a filter.On Monday, May 16, 2011 at 10:31 PM, Tim Scott wrote: > > > > Your example of retrieving documents based on a boolean is supported > > > > > in ES by filters. My app depends on using filters to narrow searches, > > > and I have not had any trouble (although I'm only in alpha). > > > > > If you only ever want filter (and not search indexed data), you'd > > > probably be better off with one of the many other NoSql alternatives, > > > IMO. > > > > > On May 16, 11:06 am, Ville [villematti.toivo...@gmail.com](mailto:villematti.toivo...@gmail.com) wrote: > > > > > > > I'm planning to use ES as a storage for JSON documents. The idea is to > > > > reference each document with index/type/id and perform search queries > > > > when needed. > > > > > > I've heard some concerns that Lucene based indexes are not reliable > > > > and that some documents might be missed. Is this still a valid > > > > concern? E.g. if I were to store items like this > > > > {..."done":"true"/"false"...}, is it possible that when searching for > > > > documents where done=false some documents are never received by the > > > > client? ---
**Author:** ![Berkay\_Mollamustafao](https://avatars.discourse-cdn.com/v4/letter/b/22d042/32.png) [@Berkay\_Mollamustafao](https://discuss.elastic.co/u/Berkay_Mollamustafao)\ **Post date:** [May 16, 2011, 9:32pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/7 "2011-05-16T21:32:57Z")
We've been using lucene based data store (even before ES with Compass) for many years in production and never had that type of problem. I don't see any reason why lucene index would be less reliable. In fact, I'd say it's a lot more reliable than most of the NoSql solutions out there. You can take the data out and browse outside to see what's in the index, etc. This type of confusion may be due to ES not updating the index right away (though it is written and safe). This is a key difference ES has and can have impact when you're using it as a data store rather than a search engine. When you write to an index, it's not immediately visible via search till the index is refreshed, which is executed once a sec by default. Regards, Berkay Mollamustafaoglu mberkay on yahoo, google and skype On Mon, May 16, 2011 at 5:25 PM, Ville [villematti.toivonen@gmail.com](mailto:villematti.toivonen@gmail.com)wrote: > Thank you for the quick replies. I'm going to do a lot of other kind > of queries also so I would like to use ES. > > However, the main point in my original post was the reliability of > Lucene/ES itself. My collegues have expressed concerns that Lucene > does not always "find" what it should. To me this would seem like a > bug but I couldn't find any bug reports with google related to these > concerns. > > Based on your responses above, it seems like that you consider ES > reliable in terms of correctness. My concern could be summarized as > "Is ES less reliable for indexing than e.g. native MySQL indexes?". I > guess not? > > On 16 touko, 22:51, Shay Banon [shay.ba...@elasticsearch.com](mailto:shay.ba...@elasticsearch.com) wrote: > > > Only filtering on boolean queries is supported by constant\_score, or you > > can also use things like term _query_ to query it, not necessarily a > > filter.On Monday, May 16, 2011 at 10:31 PM, Tim Scott wrote: > > > > Your example of retrieving documents based on a boolean is supported > > > > > in ES by filters. My app depends on using filters to narrow searches, > > > and I have not had any trouble (although I'm only in alpha). > > > > > If you only ever want filter (and not search indexed data), you'd > > > probably be better off with one of the many other NoSql alternatives, > > > IMO. > > > > > On May 16, 11:06 am, Ville [villematti.toivo...@gmail.com](mailto:villematti.toivo...@gmail.com) wrote: > > > > > > > I'm planning to use ES as a storage for JSON documents. The idea is > > > > to > > > > reference each document with index/type/id and perform search queries > > > > when needed. > > > > > > I've heard some concerns that Lucene based indexes are not reliable > > > > and that some documents might be missed. Is this still a valid > > > > concern? E.g. if I were to store items like this > > > > {..."done":"true"/"false"...}, is it possible that when searching for > > > > documents where done=false some documents are never received by the > > > > client? ---
**Author:** ![dpilato](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dpilato/32/2449_2.png) [@dpilato](https://discuss.elastic.co/u/dpilato)\ **Post date:** [May 17, 2011, 7:31am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/8 "2011-05-17T07:31:18Z")
I'm planning to use ES as a storage for JSON documents. The idea is to reference each document with index/type/id and perform search queries when needed. I was thinking about the same idea few weeks ago because ES is really easy to setup and is powerful in managing automatically shards on many instances. Then I tried to use CouchDB as the primary document storage and activate a river to ES. Really easy to setup. Really fast also to push documents into CouchDB. CouchDB allows also to add attachements (such as PDF, XML, oOo, ...) to JSON documents. I don't know at this time how ES can handle (or not) PDF files... It could be a nice design for your own needs. So is there any reason for not using CouchDB as a storage for JSON docs ? Cheers, David. ---
**Author:** ![Ville](https://avatars.discourse-cdn.com/v4/letter/v/dec6dc/32.png) [@Ville](https://discuss.elastic.co/u/Ville)\ **Post date:** [May 17, 2011, 11:36am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/9 "2011-05-17T11:36:25Z")
I want something fast, masterless and fault-tolerant with search. I've been considering Riak (nice but Search in beta), Cassandra (bloated?, tedious indexing) and ES. CouchDB could basically work for me but I've noticed that there are people not happy with its stability, see [http://labs.linkfluence.net/nosql/2011/03/07/moving\_from\_couchdb\_to\_riak.html](http://labs.linkfluence.net/nosql/2011/03/07/moving_from_couchdb_to_riak.html) I'm only going to store JSON. Do you think CouchDB+ES would be better approach instead of just ES? I don't know CouchDB that well so what would be the advantages of using it also? On May 17, 10:31 am, dpilato [david.pil...@douane.finances.gouv.fr](mailto:david.pil...@douane.finances.gouv.fr) wrote: > Ville wrote: > > > I'm planning to use ES as a storage for JSON documents. The idea is to > > reference each document with index/type/id and perform search queries > > when needed. > > I was thinking about the same idea few weeks ago because ES is really easy > to setup and is powerful in managing automatically shards on many instances. > > Then I tried to use CouchDB as the primary document storage and activate a > river to ES. > Really easy to setup. Really fast also to push documents into CouchDB. > CouchDB allows also to add attachements (such as PDF, XML, oOo, ...) to JSON > documents. I don't know at this time how ES can handle (or not) PDF files... > > It could be a nice design for your own needs. > So is there any reason for not using CouchDB as a storage for JSON docs ? > > Cheers, > David. > > -- > View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-as-a-da](http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-as-a-da)... > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [May 17, 2011, 1:11pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/10 "2011-05-17T13:11:11Z")
FWIW, we have been using ES as our primary datastore on two very large commercial projects for over a year now. I started with CouchDB and Hazelcast for a caching layer, but CouchDB (this was version 0.9 IIRC) and didn't have a sharding story and was difficult to deploy in a cloud environment. Plus it was sometimes inflating storage up to 10x the data we were storing. We were also relying on Elastic Search to provide more sophisticated query operations anyway. So, it seemed like a bad idea to store our data in CouchDB _and_ Elastic Search, so we chose to drop CouchDB. When coupled with Hazelcast (distributed cache + transaction support), the two couldn't work better together. Full disclosure: until ES reaches 1.0 status, I also store all of our persistable objects in Amazon's SimpleDB "just in case". If my S3 gateway is trashed, I can reindex straight from SimpleDB. \*Jim Cook \* [jcook@tracermedia.com](mailto:jcook@tracermedia.com) _tracermedia interactive [http://www.tracermedia.com/](http://www.tracermedia.com/)_ 780 King Ave. #106 Columbus, OH 43212 phone: (614) 298-0774 fax: (614) 298-0776 cell: (234) 738-2492 On Tue, May 17, 2011 at 7:36 AM, Ville [villematti.toivonen@gmail.com](mailto:villematti.toivonen@gmail.com)wrote: > I want something fast, masterless and fault-tolerant with search. I've > been considering Riak (nice but Search in beta), Cassandra (bloated?, > tedious indexing) and ES. > > CouchDB could basically work for me but I've noticed that there are > people not happy with its stability, see > > [http://labs.linkfluence.net/nosql/2011/03/07/moving\_from\_couchdb\_to\_riak.html](http://labs.linkfluence.net/nosql/2011/03/07/moving_from_couchdb_to_riak.html) > > I'm only going to store JSON. Do you think CouchDB+ES would be better > approach instead of just ES? I don't know CouchDB that well so what > would be the advantages of using it also? > > On May 17, 10:31 am, dpilato [david.pil...@douane.finances.gouv.fr](mailto:david.pil...@douane.finances.gouv.fr) > wrote: > > > Ville wrote: > > > > > I'm planning to use ES as a storage for JSON documents. The idea is to > > > reference each document with index/type/id and perform search queries > > > when needed. > > > > I was thinking about the same idea few weeks ago because ES is really > > easy > > to setup and is powerful in managing automatically shards on many > > instances. > > > > Then I tried to use CouchDB as the primary document storage and activate > > a > > river to ES. > > Really easy to setup. Really fast also to push documents into CouchDB. > > CouchDB allows also to add attachements (such as PDF, XML, oOo, ...) to > > JSON > > documents. I don't know at this time how ES can handle (or not) PDF > > files... > > > > It could be a nice design for your own needs. > > So is there any reason for not using CouchDB as a storage for JSON docs ? > > > > Cheers, > > David. > > > > -- > > View this message in context: > > [http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-as-a-da](http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-as-a-da)... > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![karmi](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/karmi/32/44951_2.png) [@karmi](https://discuss.elastic.co/u/karmi)\ **Post date:** [May 17, 2011, 1:56pm UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/11 "2011-05-17T13:56:41Z")
We came to the same conclusion as Jim. Once we are 100% sure that we can recover the data from gateway after _any_ crash, incident, etc, we will drop CouchDB as the storage layer. ES is better in almost any other aspect (space requirements, native clustering, etc). Karel On May 17, 3:11 pm, James Cook [jc...@tracermedia.com](mailto:jc...@tracermedia.com) wrote: > FWIW, we have been using ES as our primary datastore on two very large > commercial projects for over a year now. I started with CouchDB and > Hazelcast for a caching layer, but CouchDB (this was version 0.9 IIRC) and > didn't have a sharding story and was difficult to deploy in a cloud > environment. Plus it was sometimes inflating storage up to 10x the data we > were storing. We were also relying on Elastic Search to provide more > sophisticated query operations anyway. > > So, it seemed like a bad idea to store our data in CouchDB _and_ Elastic > Search, so we chose to drop CouchDB. When coupled with Hazelcast > (distributed cache + transaction support), the two couldn't work better > together. > > Full disclosure: until ES reaches 1.0 status, I also store all of our > persistable objects in Amazon's SimpleDB "just in case". If my S3 gateway is > trashed, I can reindex straight from SimpleDB. > > \*Jim Cook > \* > [jc...@tracermedia.com](mailto:jc...@tracermedia.com) > > _tracermedia interactive [http://www.tracermedia.com/](http://www.tracermedia.com/)_ > 780 King Ave. #106 > Columbus, OH 43212 > > phone: (614) 298-0774 > fax: (614) 298-0776 > cell: (234) 738-2492 > > On Tue, May 17, 2011 at 7:36 AM, Ville [villematti.toivo...@gmail.com](mailto:villematti.toivo...@gmail.com)wrote: > > > I want something fast, masterless and fault-tolerant with search. I've > > been considering Riak (nice but Search in beta), Cassandra (bloated?, > > tedious indexing) and ES. > > > CouchDB could basically work for me but I've noticed that there are > > people not happy with its stability, see > > > [http://labs.linkfluence.net/nosql/2011/03/07/moving\_from\_couchdb\_to\_r](http://labs.linkfluence.net/nosql/2011/03/07/moving_from_couchdb_to_r)... > > > I'm only going to store JSON. Do you think CouchDB+ES would be better > > approach instead of just ES? I don't know CouchDB that well so what > > would be the advantages of using it also? > > > On May 17, 10:31 am, dpilato [david.pil...@douane.finances.gouv.fr](mailto:david.pil...@douane.finances.gouv.fr) > > wrote: > > > > > Ville wrote: > > > > > I'm planning to use ES as a storage for JSON documents. The idea is to > > > > reference each document with index/type/id and perform search queries > > > > when needed. > > > > I was thinking about the same idea few weeks ago because ES is really > > > easy > > > to setup and is powerful in managing automatically shards on many > > > instances. > > > > Then I tried to use CouchDB as the primary document storage and activate > > > a > > > river to ES. > > > Really easy to setup. Really fast also to push documents into CouchDB. > > > CouchDB allows also to add attachements (such as PDF, XML, oOo, ...) to > > > JSON > > > documents. I don't know at this time how ES can handle (or not) PDF > > > files... > > > > It could be a nice design for your own needs. > > > So is there any reason for not using CouchDB as a storage for JSON docs ? > > > > Cheers, > > > David. > > > > -- > > > View this message in context: > > > [http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-as-a-da](http://elasticsearch-users.115913.n3.nabble.com/ElasticSearch-as-a-da)... > > > Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:05am UTC](https://discuss.elastic.co/t/elasticsearch-as-a-database/4414/12 "2017-07-06T04:05:59Z")