# Using ElasticSearch as Primary Data Store **URL:** \ **Category:** Elasticsearch\ **Created:** [December 10, 2013, 12:28pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790 "2013-12-10T12:28:54Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![vaidik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaidik/32/1134_2.png) [@vaidik](https://discuss.elastic.co/u/vaidik)\ **Post date:** [December 10, 2013, 12:28pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/1 "2013-12-10T12:28:54Z")
Hi Folks, I am working on a project where we have the following specifications: 1. Collect events at the rate of about 50-100 events/second creating almost new documents for each event. 2. We are going to query and perform multiple terms facets on that data. 3. We are not going to be performing Full Text Searches. I have played around with Elasticsearch and it solves the purpose from application point-of-view. We are able to solve all our application requirements. We would like to use just one data store where the entire data resides. And I know Elasticsearch can be used as a primary data store. But, with this scale of documents, I wonder if anyone is using Elasticsearch as their primary data store and not storing data anywhere else at all for purposes like recovery, checking accuracy, etc. I came across a case-study on [Elasticsearch.com](http://Elasticsearch.com) which mentions DataDog is using Elasticsearch as the only datastore and they moved from Postgres to ES. But the case-study itself doesn't have more specific information on the challenges and what might one be ready for when you want to have Elasticsearch as the single source of truth and how to prepare for _no data loss at all_ if something goes wrong in the ES cluster. What scenarios must one be prepared for? What else? Are there are any stories around this use-case that might help us? Would be glad to be pointed in the right direction or get some advices. Thanks, Vaidik Kapoor vaidikkapoor.info -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACWtv5nnSxmadfDz3JQ%3De3KH%3D0n6xqN\_qPCv1NjB%2BmjmKTnf4Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACWtv5nnSxmadfDz3JQ%3De3KH%3D0n6xqN_qPCv1NjB%2BmjmKTnf4Q%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![radu\_gheorghe](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radu_gheorghe/32/556_2.png) [@radu\_gheorghe](https://discuss.elastic.co/u/radu_gheorghe)\ **Post date:** [December 10, 2013, 1:15pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/2 "2013-12-10T13:15:20Z")
Hello Vaidik, I think ES is pretty safe to use as a primary data store: if a node goes down and you have replicas it will continue to function and so on. That said, if something goes seriously wrong in your cluster and your data becomes corrupted across all the replicas or something similarly tragic, you can go for backups. I guess this is valid for all data stores. If you need backups, you can look at the snapshot-restore API[http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)which just came out with 1.0.0 beta 2 [http://www.elasticsearch.org/blog/1-0-0-beta2-released/](http://www.elasticsearch.org/blog/1-0-0-beta2-released/). On Tue, Dec 10, 2013 at 2:28 PM, Vaidik Kapoor [kapoor.vaidik@gmail.com](mailto:kapoor.vaidik@gmail.com)wrote: > Hi Folks, > > I am working on a project where we have the following specifications: > > 1. Collect events at the rate of about 50-100 events/second creating > almost new documents for each event. > 2. We are going to query and perform multiple terms facets on that > data. > 3. We are not going to be performing Full Text Searches. > > I have played around with Elasticsearch and it solves the purpose from > application point-of-view. We are able to solve all our application > requirements. > We would like to use just one data store where the entire data resides. > And I know Elasticsearch can be used as a primary data store. But, with > this scale of documents, I wonder if anyone is using Elasticsearch as their > primary data store and not storing data anywhere else at all for purposes > like recovery, checking accuracy, etc. > > I came across a case-study on [Elasticsearch.com](http://Elasticsearch.com) which mentions DataDog is > using Elasticsearch as the only datastore and they moved from Postgres to > ES. But the case-study itself doesn't have more specific information on the > challenges and what might one be ready for when you want to have > Elasticsearch as the single source of truth and how to prepare for _no > data loss at all_ if something goes wrong in the ES cluster. What > scenarios must one be prepared for? What else? > > Are there are any stories around this use-case that might help us? Would > be glad to be pointed in the right direction or get some advices. > > Thanks, > > Vaidik Kapoor > vaidikkapoor.info > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CACWtv5nnSxmadfDz3JQ%3De3KH%3D0n6xqN\_qPCv1NjB%2BmjmKTnf4Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACWtv5nnSxmadfDz3JQ%3De3KH%3D0n6xqN_qPCv1NjB%2BmjmKTnf4Q%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- Performance Monitoring \* Log Analytics \* Search Analytics Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_05h2Rj7rpetvEvoXTZVaqBBza7deM6mTu%3D%3D6m-hMc%2B9A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_05h2Rj7rpetvEvoXTZVaqBBza7deM6mTu%3D%3D6m-hMc%2B9A%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vaidik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaidik/32/1134_2.png) [@vaidik](https://discuss.elastic.co/u/vaidik)\ **Post date:** [December 10, 2013, 11:09pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/3 "2013-12-10T23:09:25Z")
Fair enough. That introduces me to a couple of things to explore. I shall shoot more questions on this very thread if and when I have more to ask/consult/share. Thanks for your answer. 🙂 Vaidik Kapoor vaidikkapoor.info On 10 December 2013 18:45, Radu Gheorghe [radu.gheorghe@sematext.com](mailto:radu.gheorghe@sematext.com) wrote: > Hello Vaidik, > > I think ES is pretty safe to use as a primary data store: if a node goes > down and you have replicas it will continue to function and so on. > > That said, if something goes seriously wrong in your cluster and your data > becomes corrupted across all the replicas or something similarly tragic, > you can go for backups. I guess this is valid for all data stores. > > If you need backups, you can look at the snapshot-restore API[http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html](http://www.elasticsearch.org/guide/en/elasticsearch/reference/master/modules-snapshots.html)which just came out with 1.0.0 > beta 2 [http://www.elasticsearch.org/blog/1-0-0-beta2-released/](http://www.elasticsearch.org/blog/1-0-0-beta2-released/). > > On Tue, Dec 10, 2013 at 2:28 PM, Vaidik Kapoor [kapoor.vaidik@gmail.com](mailto:kapoor.vaidik@gmail.com)wrote: > > > Hi Folks, > > > > I am working on a project where we have the following specifications: > > > > 1. Collect events at the rate of about 50-100 events/second creating > > almost new documents for each event. > > 2. We are going to query and perform multiple terms facets on that > > data. > > 3. We are not going to be performing Full Text Searches. > > > > I have played around with Elasticsearch and it solves the purpose from > > application point-of-view. We are able to solve all our application > > requirements. > > We would like to use just one data store where the entire data resides. > > And I know Elasticsearch can be used as a primary data store. But, with > > this scale of documents, I wonder if anyone is using Elasticsearch as their > > primary data store and not storing data anywhere else at all for purposes > > like recovery, checking accuracy, etc. > > > > I came across a case-study on [Elasticsearch.com](http://Elasticsearch.com) which mentions DataDog is > > using Elasticsearch as the only datastore and they moved from Postgres to > > ES. But the case-study itself doesn't have more specific information on the > > challenges and what might one be ready for when you want to have > > Elasticsearch as the single source of truth and how to prepare for _no > > data loss at all_ if something goes wrong in the ES cluster. What > > scenarios must one be prepared for? What else? > > > > Are there are any stories around this use-case that might help us? Would > > be glad to be pointed in the right direction or get some advices. > > > > Thanks, > > > > Vaidik Kapoor > > vaidikkapoor.info > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CACWtv5nnSxmadfDz3JQ%3De3KH%3D0n6xqN\_qPCv1NjB%2BmjmKTnf4Q%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACWtv5nnSxmadfDz3JQ%3De3KH%3D0n6xqN_qPCv1NjB%2BmjmKTnf4Q%40mail.gmail.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > Performance Monitoring \* Log Analytics \* Search Analytics > Solr & Elasticsearch Support \* [http://sematext.com/](http://sematext.com/) > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CAHXA0\_05h2Rj7rpetvEvoXTZVaqBBza7deM6mTu%3D%3D6m-hMc%2B9A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAHXA0_05h2Rj7rpetvEvoXTZVaqBBza7deM6mTu%3D%3D6m-hMc%2B9A%40mail.gmail.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACWtv5kzQxODbfRiNm3fUZ-xnU9Bs7BG%2BrooQdVLrHCuB3nKuQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACWtv5kzQxODbfRiNm3fUZ-xnU9Bs7BG%2BrooQdVLrHCuB3nKuQ%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Eugene\_Strokin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/eugene_strokin/32/1238_2.png) [@Eugene\_Strokin](https://discuss.elastic.co/u/Eugene_Strokin)\ **Post date:** [December 11, 2013, 2:52am UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/4 "2013-12-11T02:52:49Z")
I use ES as a primary datasource from 0.2 version. It is in production for almost 2 years. Starting from 5 shards all on the same node, to 1 replica of those 5 shards on 3 nodes. Serves about a doezen requests per seconds in average. All kind of requests, searches, filtering, sorting, faceting. I had transferring whole cluster even to different datacenters with zero down time several times. All problems I had was only because I did something wrong, but it wasn't fault of ES. So, I could say now, that ES could be used as the only data store. I'v tried several other options, like Solar - too hard to scale, Cassandra - not easy to support complex (and I wouldn't even call it complex really) data structure. Hbase on hadoop - too low level. And performance of ES is very impressive comparing to others. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![vaidik](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vaidik/32/1134_2.png) [@vaidik](https://discuss.elastic.co/u/vaidik)\ **Post date:** [December 11, 2013, 9:03am UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/5 "2013-12-11T09:03:46Z")
For our use-case, we need facets desperately. Otherwise we will have to do that in the application logic, which is not ideal and honestly a lot of work too. ES gives me that. However, with the number of documents we need to index per second (30-50 per second, and this number is going to grow with time), I wonder what do people do to make sure that: - There are least chances of data loss. You cannot flush segments to disk very quickly as that won't be optimal and to my knowledge a lot of unoptimized segments will be created if I manually use the Flush API. So what does ES do, when data has been written to translog but the operations have not been flushed and the node goes down? - If there is scope for data loss, then what does one do to detect it? Being new to ES, I am still trying to understand where is ES using JVM heap properly and how it can affect my cluster. Consider this: we have three nodes and we are indexing data in it at the rate of 30-50 docs per second. When I started the cluster, JVM heap usage was low (about 2-4% on each node). With time, that keeps on growing and stabilizes in between 81-94%. Now, in the meanwhile I am just indexing and not querying data at all from the cluster. I am using G1-GC instead of CMS-GC because CMS-GC was giving me long long pauses (about 13-17 seconds for garbage collection) which is not ideal. With G1, GC is frequent and quicker (so far I have seen about 1 second). This works but I am always concerned that the JVM Heap usage is so high and if a little more load is in the pipeline, then what will happen. Will ES be able to take it or there are chances of experiencing OutOfMemory exceptions, leading the node to go down. Obviously, this is something that I will have to test according to my use-case, but I am interested in knowing if there is someone around here who has experienced similar problems and have found the solution or a work around. After some time, GC happens so quickly that I can make out that it is affecting indexing (I am indexing using a Rabbit consumer written in Python and after every 10-20 seconds, I'd see a peak in the queue, suggesting that the consumer is not able to consume, further suggesting that the consumer is not able to quickly write to ES, leading me to assume that GC is the cause of the slow write as the CPU is busy. So: - What are/could be the reasons of such heap usage? What is ES doing with so much heap? - How can I keep that in control? Thanks Vaidik Kapoor vaidikkapoor.info On 11 December 2013 08:22, Eugene Strokin [eugene@strokin.info](mailto:eugene@strokin.info) wrote: > I use ES as a primary datasource from 0.2 version. It is in production for > almost 2 years. Starting from 5 shards all on the same node, to 1 replica > of those 5 shards on 3 nodes. Serves about a doezen requests per seconds in > average. All kind of requests, searches, filtering, sorting, faceting. I > had transferring whole cluster even to different datacenters with zero down > time several times. All problems I had was only because I did something > wrong, but it wasn't fault of ES. > So, I could say now, that ES could be used as the only data store. > I'v tried several other options, like Solar - too hard to scale, Cassandra > > - not easy to support complex (and I wouldn't even call it complex really) > data structure. Hbase on hadoop - too low level. And performance of ES is > very impressive comparing to others. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com) > . > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACWtv5mTCSmU73MFyMRWJUvbiMXzGj%3DPssLWc7rwjuD-ZAOwVg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACWtv5mTCSmU73MFyMRWJUvbiMXzGj%3DPssLWc7rwjuD-ZAOwVg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [December 11, 2013, 11:10am UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/6 "2013-12-11T11:10:46Z")
ES picks up translogs and replays them the next start after a node went down. You work heavily with facets, and I share your concerns about OOMs inducing flakiness to the whole cluster. Have you checked how your cache on the heap is used? Note that some caches are turned on by default and may interfere with your facets. You should also look into the new aggregation framework of 1.0.0.Beta2 if the new faceting is less resource consuming. Regarding the indexing, check if you have a strategy for segment merging and throttling. Setting custom values can take much pressure off the heap, especially when segments grow very large. And finally, check if you can identify the sweet spot when to add nodes, if heap usage is just getting too high. Jörg On Wed, Dec 11, 2013 at 10:03 AM, Vaidik Kapoor [kapoor.vaidik@gmail.com](mailto:kapoor.vaidik@gmail.com)wrote: > For our use-case, we need facets desperately. Otherwise we will have to do > that in the application logic, which is not ideal and honestly a lot of > work too. ES gives me that. However, with the number of documents we need > to index per second (30-50 per second, and this number is going to grow > with time), I wonder what do people do to make sure that: > > - There are least chances of data loss. You cannot flush segments to disk > very quickly as that won't be optimal and to my knowledge a lot of > unoptimized segments will be created if I manually use the Flush API. So > what does ES do, when data has been written to translog but the operations > have not been flushed and the node goes down? > - If there is scope for data loss, then what does one do to detect it? > > Being new to ES, I am still trying to understand where is ES using JVM > heap properly and how it can affect my cluster. Consider this: we have > three nodes and we are indexing data in it at the rate of 30-50 docs per > second. When I started the cluster, JVM heap usage was low (about 2-4% on > each node). With time, that keeps on growing and stabilizes in between > 81-94%. Now, in the meanwhile I am just indexing and not querying data at > all from the cluster. I am using G1-GC instead of CMS-GC because CMS-GC was > giving me long long pauses (about 13-17 seconds for garbage collection) > which is not ideal. With G1, GC is frequent and quicker (so far I have seen > about 1 second). This works but I am always concerned that the JVM Heap > usage is so high and if a little more load is in the pipeline, then what > will happen. Will ES be able to take it or there are chances of > experiencing OutOfMemory exceptions, leading the node to go down. > Obviously, this is something that I will have to test according to my > use-case, but I am interested in knowing if there is someone around here > who has experienced similar problems and have found the solution or a work > around. > > After some time, GC happens so quickly that I can make out that it is > affecting indexing (I am indexing using a Rabbit consumer written in Python > and after every 10-20 seconds, I'd see a peak in the queue, suggesting that > the consumer is not able to consume, further suggesting that the consumer > is not able to quickly write to ES, leading me to assume that GC is the > cause of the slow write as the CPU is busy. > > So: > > - What are/could be the reasons of such heap usage? What is ES doing with > so much heap? > - How can I keep that in control? > > Thanks > > Vaidik Kapoor > vaidikkapoor.info > > On 11 December 2013 08:22, Eugene Strokin [eugene@strokin.info](mailto:eugene@strokin.info) wrote: > > > I use ES as a primary datasource from 0.2 version. It is in production > > for almost 2 years. Starting from 5 shards all on the same node, to 1 > > replica of those 5 shards on 3 nodes. Serves about a doezen requests per > > seconds in average. All kind of requests, searches, filtering, sorting, > > faceting. I had transferring whole cluster even to different datacenters > > with zero down time several times. All problems I had was only because I > > did something wrong, but it wasn't fault of ES. > > So, I could say now, that ES could be used as the only data store. > > I'v tried several other options, like Solar - too hard to scale, > > Cassandra - not easy to support complex (and I wouldn't even call it > > complex really) data structure. Hbase on hadoop - too low level. And > > performance of ES is very impressive comparing to others. > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CACWtv5mTCSmU73MFyMRWJUvbiMXzGj%3DPssLWc7rwjuD-ZAOwVg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACWtv5mTCSmU73MFyMRWJUvbiMXzGj%3DPssLWc7rwjuD-ZAOwVg%40mail.gmail.com) > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoErrt6LDFDwMXKsqBmyQoVgi8hn6mDNNDXA1-u5cXt5bg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoErrt6LDFDwMXKsqBmyQoVgi8hn6mDNNDXA1-u5cXt5bg%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![davrob](https://avatars.discourse-cdn.com/v4/letter/d/d78d45/32.png) [@davrob](https://discuss.elastic.co/u/davrob)\ **Post date:** [December 11, 2013, 4:22pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/7 "2013-12-11T16:22:31Z")
Hi Jorg, So if my index files get corrupted and I restore (using the new Snapshot API) the cluster, is there a way of moving the translogs from the old cluster nodes to the new one? - David. On Wednesday, 11 December 2013 11:10:46 UTC, Jörg Prante wrote: > ES picks up translogs and replays them the next start after a node went > down. > > You work heavily with facets, and I share your concerns about OOMs > inducing flakiness to the whole cluster. > > Have you checked how your cache on the heap is used? Note that some caches > are turned on by default and may interfere with your facets. > > You should also look into the new aggregation framework of 1.0.0.Beta2 if > the new faceting is less resource consuming. > > Regarding the indexing, check if you have a strategy for segment merging > and throttling. Setting custom values can take much pressure off the heap, > especially when segments grow very large. > > And finally, check if you can identify the sweet spot when to add nodes, > if heap usage is just getting too high. > > Jörg > > On Wed, Dec 11, 2013 at 10:03 AM, Vaidik Kapoor \<[kapoor...@gmail.com](mailto:kapoor...@gmail.com)\ > > > wrote: > > > For our use-case, we need facets desperately. Otherwise we will have to > > do that in the application logic, which is not ideal and honestly a lot of > > work too. ES gives me that. However, with the number of documents we need > > to index per second (30-50 per second, and this number is going to grow > > with time), I wonder what do people do to make sure that: > > > > - There are least chances of data loss. You cannot flush segments to disk > > very quickly as that won't be optimal and to my knowledge a lot of > > unoptimized segments will be created if I manually use the Flush API. So > > what does ES do, when data has been written to translog but the operations > > have not been flushed and the node goes down? > > - If there is scope for data loss, then what does one do to detect it? > > > > Being new to ES, I am still trying to understand where is ES using JVM > > heap properly and how it can affect my cluster. Consider this: we have > > three nodes and we are indexing data in it at the rate of 30-50 docs per > > second. When I started the cluster, JVM heap usage was low (about 2-4% on > > each node). With time, that keeps on growing and stabilizes in between > > 81-94%. Now, in the meanwhile I am just indexing and not querying data at > > all from the cluster. I am using G1-GC instead of CMS-GC because CMS-GC was > > giving me long long pauses (about 13-17 seconds for garbage collection) > > which is not ideal. With G1, GC is frequent and quicker (so far I have seen > > about 1 second). This works but I am always concerned that the JVM Heap > > usage is so high and if a little more load is in the pipeline, then what > > will happen. Will ES be able to take it or there are chances of > > experiencing OutOfMemory exceptions, leading the node to go down. > > Obviously, this is something that I will have to test according to my > > use-case, but I am interested in knowing if there is someone around here > > who has experienced similar problems and have found the solution or a work > > around. > > > > After some time, GC happens so quickly that I can make out that it is > > affecting indexing (I am indexing using a Rabbit consumer written in Python > > and after every 10-20 seconds, I'd see a peak in the queue, suggesting that > > the consumer is not able to consume, further suggesting that the consumer > > is not able to quickly write to ES, leading me to assume that GC is the > > cause of the slow write as the CPU is busy. > > > > So: > > > > - What are/could be the reasons of such heap usage? What is ES doing with > > so much heap? > > - How can I keep that in control? > > > > Thanks > > > > Vaidik Kapoor > > vaidikkapoor.info > > > > On 11 December 2013 08:22, Eugene Strokin \ > > > > > wrote: > > > > > I use ES as a primary datasource from 0.2 version. It is in production > > > for almost 2 years. Starting from 5 shards all on the same node, to 1 > > > replica of those 5 shards on 3 nodes. Serves about a doezen requests per > > > seconds in average. All kind of requests, searches, filtering, sorting, > > > faceting. I had transferring whole cluster even to different datacenters > > > with zero down time several times. All problems I had was only because I > > > did something wrong, but it wasn't fault of ES. > > > So, I could say now, that ES could be used as the only data store. > > > I'v tried several other options, like Solar - too hard to scale, > > > Cassandra - not easy to support complex (and I wouldn't even call it > > > complex really) data structure. Hbase on hadoop - too low level. And > > > performance of ES is very impressive comparing to others. > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > > To view this discussion on the web visit > > > [https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9%40googlegroups.com) > > > . > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com) \. > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/CACWtv5mTCSmU73MFyMRWJUvbiMXzGj%3DPssLWc7rwjuD-ZAOwVg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACWtv5mTCSmU73MFyMRWJUvbiMXzGj%3DPssLWc7rwjuD-ZAOwVg%40mail.gmail.com) > > . > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/af52a997-6880-41ca-afcc-c9ac47a9098c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/af52a997-6880-41ca-afcc-c9ac47a9098c%40googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![mattweber](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mattweber/32/44940_2.png) [@mattweber](https://discuss.elastic.co/u/mattweber)\ **Post date:** [December 11, 2013, 4:38pm UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/8 "2013-12-11T16:38:18Z")
For those of you concerned with OOM and heap issues due to facets and/or caching I would consider waiting for 1.0 which will have: Field Data Circuit breaker to limit how much memory is used. > > > This adds the field data circuit breaker, which is used to estimate > the amount o…f memory required to load field data before loading it. It > then raises a CircuitBreakingException if the limit is exceeded. > > It is configured with two parameters: > > \`indices.fielddata.cache.breaker.limit\` - the maximum number of bytes > of field data to be loaded before circuit breaking. Defaults to > \`indices.fielddata.cache.size\` if set, unbounded otherwise. > > \`indices.fielddata.cache.breaker.overhead\` - a contast for all field > data estimations to be multiplied with before aggregation. Defaults to > 1.03. > > Both settings can be configured dynamically using the cluster update > settings API. Disk based fielddata/docvalues > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. The docvalues is in beta2, I imagine the circuit breaker will be in the next. On top of that there has been a lot of work going on to reduce the number of objects created so there will be less GC's in general. Thanks, Matt Weber On Wed, Dec 11, 2013 at 8:22 AM, davrob2 [daviroberts@gmail.com](mailto:daviroberts@gmail.com) wrote: > Hi Jorg, > > So if my index files get corrupted and I restore (using the new Snapshot > API) the cluster, is there a way of moving the translogs from the old > cluster nodes to the new one? > > - David. > > On Wednesday, 11 December 2013 11:10:46 UTC, Jörg Prante wrote: > > > ES picks up translogs and replays them the next start after a node went > > down. > > > > You work heavily with facets, and I share your concerns about OOMs > > inducing flakiness to the whole cluster. > > > > Have you checked how your cache on the heap is used? Note that some > > caches are turned on by default and may interfere with your facets. > > > > You should also look into the new aggregation framework of 1.0.0.Beta2 if > > the new faceting is less resource consuming. > > > > Regarding the indexing, check if you have a strategy for segment merging > > and throttling. Setting custom values can take much pressure off the heap, > > especially when segments grow very large. > > > > And finally, check if you can identify the sweet spot when to add nodes, > > if heap usage is just getting too high. > > > > Jörg > > > > On Wed, Dec 11, 2013 at 10:03 AM, Vaidik Kapoor [kapoor...@gmail.com](mailto:kapoor...@gmail.com)wrote: > > > > > For our use-case, we need facets desperately. Otherwise we will have to > > > do that in the application logic, which is not ideal and honestly a lot of > > > work too. ES gives me that. However, with the number of documents we need > > > to index per second (30-50 per second, and this number is going to grow > > > with time), I wonder what do people do to make sure that: > > > > > > - There are least chances of data loss. You cannot flush segments to > > > disk very quickly as that won't be optimal and to my knowledge a lot of > > > unoptimized segments will be created if I manually use the Flush API. So > > > what does ES do, when data has been written to translog but the operations > > > have not been flushed and the node goes down? > > > - If there is scope for data loss, then what does one do to detect it? > > > > > > Being new to ES, I am still trying to understand where is ES using JVM > > > heap properly and how it can affect my cluster. Consider this: we have > > > three nodes and we are indexing data in it at the rate of 30-50 docs per > > > second. When I started the cluster, JVM heap usage was low (about 2-4% on > > > each node). With time, that keeps on growing and stabilizes in between > > > 81-94%. Now, in the meanwhile I am just indexing and not querying data at > > > all from the cluster. I am using G1-GC instead of CMS-GC because CMS-GC was > > > giving me long long pauses (about 13-17 seconds for garbage collection) > > > which is not ideal. With G1, GC is frequent and quicker (so far I have seen > > > about 1 second). This works but I am always concerned that the JVM Heap > > > usage is so high and if a little more load is in the pipeline, then what > > > will happen. Will ES be able to take it or there are chances of > > > experiencing OutOfMemory exceptions, leading the node to go down. > > > Obviously, this is something that I will have to test according to my > > > use-case, but I am interested in knowing if there is someone around here > > > who has experienced similar problems and have found the solution or a work > > > around. > > > > > > After some time, GC happens so quickly that I can make out that it is > > > affecting indexing (I am indexing using a Rabbit consumer written in Python > > > and after every 10-20 seconds, I'd see a peak in the queue, suggesting that > > > the consumer is not able to consume, further suggesting that the consumer > > > is not able to quickly write to ES, leading me to assume that GC is the > > > cause of the slow write as the CPU is busy. > > > > > > So: > > > > > > - What are/could be the reasons of such heap usage? What is ES doing > > > with so much heap? > > > - How can I keep that in control? > > > > > > Thanks > > > > > > Vaidik Kapoor > > > vaidikkapoor.info > > > > > > On 11 December 2013 08:22, Eugene Strokin [eug...@strokin.info](mailto:eug...@strokin.info) wrote: > > > > > > > I use ES as a primary datasource from 0.2 version. It is in production > > > > for almost 2 years. Starting from 5 shards all on the same node, to 1 > > > > replica of those 5 shards on 3 nodes. Serves about a doezen requests per > > > > seconds in average. All kind of requests, searches, filtering, sorting, > > > > faceting. I had transferring whole cluster even to different datacenters > > > > with zero down time several times. All problems I had was only because I > > > > did something wrong, but it wasn't fault of ES. > > > > So, I could say now, that ES could be used as the only data store. > > > > I'v tried several other options, like Solar - too hard to scale, > > > > Cassandra - not easy to support complex (and I wouldn't even call it > > > > complex really) data structure. Hbase on hadoop - too low level. And > > > > performance of ES is very impressive comparing to others. > > > > > > > > -- > > > > You received this message because you are subscribed to the Google > > > > Groups "elasticsearch" group. > > > > To unsubscribe from this group and stop receiving emails from it, send > > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > > msgid/elasticsearch/791b3323-fcf9-47fb-b874-e7e7b9feb7c9% > > > > [40googlegroups.com](http://40googlegroups.com). > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > > > -- > > > You received this message because you are subscribed to the Google > > > Groups "elasticsearch" group. > > > To unsubscribe from this group and stop receiving emails from it, send > > > an email to [elasticsearc...@googlegroups.com](mailto:elasticsearc...@googlegroups.com). > > > To view this discussion on the web visit [https://groups.google.com/d/](https://groups.google.com/d/) > > > msgid/elasticsearch/CACWtv5mTCSmU73MFyMRWJUvbiMXzG > > > j%3DPssLWc7rwjuD-ZAOwVg%[40mail.gmail.com](http://40mail.gmail.com). > > > > > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/af52a997-6880-41ca-afcc-c9ac47a9098c%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/af52a997-6880-41ca-afcc-c9ac47a9098c%40googlegroups.com) > > . > > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoAvO6085KeUo0yf-mBQg0qgrF86g-cZL%2Bb-EM-DhDwFiA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAJ3KEoAvO6085KeUo0yf-mBQg0qgrF86g-cZL%2Bb-EM-DhDwFiA%40mail.gmail.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 2:01am UTC](https://discuss.elastic.co/t/using-elasticsearch-as-primary-data-store/14790/9 "2017-07-06T02:01:56Z")