# Using ES as our primary and only datastore **URL:** \ **Category:** Elasticsearch\ **Created:** [June 23, 2011, 3:55pm UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694 "2011-06-23T15:55:12Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Spring\_Ninja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spring_ninja/32/2719_2.png) [@Spring\_Ninja](https://discuss.elastic.co/u/Spring_Ninja)\ **Post date:** [June 23, 2011, 3:55pm UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/1 "2011-06-23T15:55:12Z")
I have read through some threads where people mention that their only datastore is ES. From what I have read it can be an option. However, can you point out some drawbacks in doing so? My main no-go is transaction support. I have another project using Google App Engine and have an app that does not use transactions. It can be done but requires more thinking. I am currently syncing ES with MySQL, including transaction support. However, I would love to let go of MySQL and not have to worry about scaling MySQL. Thanks! Remy ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 23, 2011, 4:41pm UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/2 "2011-06-23T16:41:59Z")
Hi Remy, I'm not answering but I'm asking a question about your case. How do you sync MySql and ES ? thanks David 😉 Le 23 juin 2011 à 17:55, Spring Ninja [remy.gendron@ingeno.ca](mailto:remy.gendron@ingeno.ca) a écrit : > I have read through some threads where people mention that their only > datastore is ES. From what I have read it can be an option. > > However, can you point out some drawbacks in doing so? My main no-go > is transaction support. I have another project using Google App Engine > and have an app that does not use transactions. It can be done but > requires more thinking. > > I am currently syncing ES with MySQL, including transaction support. > However, I would love to let go of MySQL and not have to worry about > scaling MySQL. > > Thanks! > > Remy ---
**Author:** ![ppearcy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ppearcy/32/980_2.png) [@ppearcy](https://discuss.elastic.co/u/ppearcy)\ **Post date:** [June 23, 2011, 7:23pm UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/3 "2011-06-23T19:23:08Z")
Hey, This discussion should answer most of your questions regarding ES as the primary data store: [http://groups.google.com/a/elasticsearch.com/group/users/browse\_thread/thread/f8ff072a7039292d/f90501e0861b4e99#f90501e0861b4e99](http://groups.google.com/a/elasticsearch.com/group/users/browse_thread/thread/f8ff072a7039292d/f90501e0861b4e99#f90501e0861b4e99) Best Regards, Paul On Jun 23, 10:41 am, David Pilato [da...@pilato.fr](mailto:da...@pilato.fr) wrote: > Hi Remy, > > I'm not answering but I'm asking a question about your case. > How do you sync MySql and ES ? > > thanks > David 😉 > > Le 23 juin 2011 à 17:55, Spring Ninja [remy.gend...@ingeno.ca](mailto:remy.gend...@ingeno.ca) a écrit : > > > I have read through some threads where people mention that their only > > datastore is ES. From what I have read it can be an option. > > > However, can you point out some drawbacks in doing so? My main no-go > > is transaction support. I have another project using Google App Engine > > and have an app that does not use transactions. It can be done but > > requires more thinking. > > > I am currently syncing ES with MySQL, including transaction support. > > However, I would love to let go of MySQL and not have to worry about > > scaling MySQL. > > > Thanks! > > > Remy ---
**Author:** ![Remy\_Gendron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/remy_gendron/32/2719_2.png) [@Remy\_Gendron](https://discuss.elastic.co/u/Remy_Gendron)\ **Post date:** [June 24, 2011, 3:54am UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/4 "2011-06-24T03:54:01Z")
Well, just hooking into the lifecycle events from our service layer and replicating the cruds to ES. It's a lot like Hibernate Search for those who know. ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [June 24, 2011, 4:45am UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/5 "2011-06-24T04:45:37Z")
Thanks. That's the way I did it also. At first, I was thinking of not modifying the service layer. So I tried to use hibernate listeners but it didn't work really fine because when you update a child entity only, even if you ask to hibernate to merge from the parent entity, the listener is only called for the child entity. I didn't find an easy way to do that. Thanks David 😉 Le 24 juin 2011 à 05:54, Remy Gendron [remy@arrova.ca](mailto:remy@arrova.ca) a écrit : > Well, just hooking into the lifecycle events from our service layer > and replicating the cruds to ES. It's a lot like Hibernate Search for > those who know. ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 27, 2011, 5:19pm UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/6 "2011-06-27T17:19:47Z")
Some of the drawbacks of using ES as a datastore are: 1. Lack of transactional support (as you mention). We use Hazelcast as a memcached layer between our application and ES. Hazelcast supports distributed transactions. While not supporting commit and rollback upon a write failure, it does give us the ability to commit multiple writes as a single unit of work. 2. No snapshotting of data. I miss this greatly for peace of mind. You have to code your own solutions to be able to recover from corruption of the Lucene indexes, as well as when ES introduces a change that requires a reindexing of all the data. A shared gateway can help, but you will have to pause indexing/flushing while making a backup of the repository. 3. Near real time behavior is hard for most who come from DB background. You can't insert a record then query for that record without issuing a refresh call or delaying long enough to ensure the record has been indexed. 4. No SQL support. It goes without saying, but this has an impact in such there are no tools which allow you to manipulate data once it is in the repository. Perhaps these will come in time. Those are the big ones that I have noticed. \*Jim Cook \* [jcook@tracermedia.com](mailto:jcook@tracermedia.com) _tracermedia interactive [http://www.tracermedia.com/](http://www.tracermedia.com/)_ 780 King Ave. #106 Columbus, OH 43212 phone: (614) 298-0774 fax: (614) 298-0776 cell: (234) 738-2492 On Thu, Jun 23, 2011 at 11:55 AM, Spring Ninja [remy.gendron@ingeno.ca](mailto:remy.gendron@ingeno.ca)wrote: > I have read through some threads where people mention that their only > datastore is ES. From what I have read it can be an option. > > However, can you point out some drawbacks in doing so? My main no-go > is transaction support. I have another project using Google App Engine > and have an app that does not use transactions. It can be done but > requires more thinking. > > I am currently syncing ES with MySQL, including transaction support. > However, I would love to let go of MySQL and not have to worry about > scaling MySQL. > > Thanks! > > Remy ---
**Author:** ![James\_Cook](https://avatars.discourse-cdn.com/v4/letter/j/898d66/32.png) [@James\_Cook](https://discuss.elastic.co/u/James_Cook)\ **Post date:** [June 27, 2011, 6:01pm UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/7 "2011-06-27T18:01:37Z")
I suppose #3 below is mitigated by this new feature: > > > Realtime GET support allows to get a document once indexed regardless of the "re…fresh rate" of the index. It is enabled by default. > > In order to disable realtime GET, one can either set \`realtime\` parameter to \`false\`, or globally default it to by setting the \`action.get.realtime\` to \`false\` in the node configuration. > \# Realtime GET and stored fields > > When getting a document, one can specify \`fields\` to fetch from it. They will, when possible, be fetched as stored fields (fields mapped as stored in the mapping). When using realtime GET, there is no notion of stored fields (at least for a period of time, basically, until the next flush), so they will be extracted from the source itself (note, even if source is not enabled). It is a good practice to assume that the fields will be loaded from source when using realtime GET, even if the fields are stored. -- jim On Mon, Jun 27, 2011 at 1:19 PM, James Cook [jcook@tracermedia.com](mailto:jcook@tracermedia.com) wrote: > Some of the drawbacks of using ES as a datastore are: > > 1. Lack of transactional support (as you mention). We use Hazelcast as > a memcached layer between our application and ES. Hazelcast supports > distributed transactions. While not supporting commit and rollback upon a > write failure, it does give us the ability to commit multiple writes as a > single unit of work. > 2. No snapshotting of data. I miss this greatly for peace of mind. You > have to code your own solutions to be able to recover from corruption of the > Lucene indexes, as well as when ES introduces a change that requires a > reindexing of all the data. A shared gateway can help, but you will have to > pause indexing/flushing while making a backup of the repository. > 3. Near real time behavior is hard for most who come from DB > background. You can't insert a record then query for that record without > issuing a refresh call or delaying long enough to ensure the record has been > indexed. > 4. No SQL support. It goes without saying, but this has an impact in > such there are no tools which allow you to manipulate data once it is in the > repository. Perhaps these will come in time. > > Those are the big ones that I have noticed. > > \*Jim Cook > \* > [jcook@tracermedia.com](mailto:jcook@tracermedia.com) > > _tracermedia interactive [http://www.tracermedia.com/](http://www.tracermedia.com/)_ > 780 King Ave. #106 > Columbus, OH 43212 > > phone: (614) 298-0774 > fax: (614) 298-0776 > cell: (234) 738-2492 > > On Thu, Jun 23, 2011 at 11:55 AM, Spring Ninja [remy.gendron@ingeno.ca](mailto:remy.gendron@ingeno.ca)wrote: > > > I have read through some threads where people mention that their only > > datastore is ES. From what I have read it can be an option. > > > > However, can you point out some drawbacks in doing so? My main no-go > > is transaction support. I have another project using Google App Engine > > and have an app that does not use transactions. It can be done but > > requires more thinking. > > > > I am currently syncing ES with MySQL, including transaction support. > > However, I would love to let go of MySQL and not have to worry about > > scaling MySQL. > > > > Thanks! > > > > Remy ---
**Author:** ![Remy\_Gendron](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/remy_gendron/32/2719_2.png) [@Remy\_Gendron](https://discuss.elastic.co/u/Remy_Gendron)\ **Post date:** [June 28, 2011, 2:28am UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/8 "2011-06-28T02:28:05Z")
Great summary James, thanks! Shay, is this something that is considered on the long term roadmap, things such as index migration when a new release comes out, transactions, full restore, etc? I guess that if Lucene isn't meant to be a datastore, it would be hard for ES to provide this... On Jun 27, 2:01 pm, James Cook [jc...@tracermedia.com](mailto:jc...@tracermedia.com) wrote: > I suppose #3 below is mitigated by this new feature:[Realtime GET · Issue #1060 · elastic/elasticsearch · GitHub](https://github.com/elasticsearch/elasticsearch/issues/1060) > > -- jim > > On Mon, Jun 27, 2011 at 1:19 PM, James Cook [jc...@tracermedia.com](mailto:jc...@tracermedia.com) wrote: > > > Some of the drawbacks of using ES as a datastore are: > > > 1. Lack of transactional support (as you mention). We use Hazelcast as > > a memcached layer between our application and ES. Hazelcast supports > > distributed transactions. While not supporting commit and rollback upon a > > write failure, it does give us the ability to commit multiple writes as a > > single unit of work. > > 2. No snapshotting of data. I miss this greatly for peace of mind. You > > have to code your own solutions to be able to recover from corruption of the > > Lucene indexes, as well as when ES introduces a change that requires a > > reindexing of all the data. A shared gateway can help, but you will have to > > pause indexing/flushing while making a backup of the repository. > > 3. Near real time behavior is hard for most who come from DB > > background. You can't insert a record then query for that record without > > issuing a refresh call or delaying long enough to ensure the record has been > > indexed. > > 4. No SQL support. It goes without saying, but this has an impact in > > such there are no tools which allow you to manipulate data once it is in the > > repository. Perhaps these will come in time. > > > Those are the big ones that I have noticed. > > > \*Jim Cook > > \* > > [jc...@tracermedia.com](mailto:jc...@tracermedia.com) > > > _tracermedia interactive [http://www.tracermedia.com/](http://www.tracermedia.com/)_ > > 780 King Ave. #106 > > Columbus, OH 43212 > > > phone: (614) 298-0774 > > fax: (614) 298-0776 > > cell: (234) 738-2492 > > > On Thu, Jun 23, 2011 at 11:55 AM, Spring Ninja [remy.gend...@ingeno.ca](mailto:remy.gend...@ingeno.ca)wrote: > > > > I have read through some threads where people mention that their only > > > datastore is ES. From what I have read it can be an option. > > > > However, can you point out some drawbacks in doing so? My main no-go > > > is transaction support. I have another project using Google App Engine > > > and have an app that does not use transactions. It can be done but > > > requires more thinking. > > > > I am currently syncing ES with MySQL, including transaction support. > > > However, I would love to let go of MySQL and not have to worry about > > > scaling MySQL. > > > > Thanks! > > > > Remy ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 4:02am UTC](https://discuss.elastic.co/t/using-es-as-our-primary-and-only-datastore/4694/9 "2017-07-06T04:02:25Z")