# Using ES as a primary datastore **URL:** \ **Category:** Elasticsearch\ **Created:** [September 17, 2014, 10:41am UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839 "2014-09-17T10:41:55Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![P\_Suman](https://avatars.discourse-cdn.com/v4/letter/p/3bc359/32.png) [@P\_Suman](https://discuss.elastic.co/u/P_Suman)\ **Post date:** [September 17, 2014, 10:41am UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/1 "2014-09-17T10:41:55Z")
Hello, We are planning to use ES as a primary datastore. Here is my usecase We receive a million transactions per day (all are inserts). Each transaction is around 500KB size, transaction has 10 fields we should be able to search on all 10 fields. We want to keep around 1 yr worth of data, this comes around 180TB Can you please let me know any problems that might arise if i use elastic search as the primary datastore. Regards, Suman -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f52b61e2-0955-4e79-8bb8-61c9428c67d1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f52b61e2-0955-4e79-8bb8-61c9428c67d1%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [September 17, 2014, 11:58am UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/2 "2014-09-17T11:58:07Z")
That's a lot of data, do you have a big budget, automation, monitoring? Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 17 September 2014 20:41, P Suman [papanaboina.suman@gmail.com](mailto:papanaboina.suman@gmail.com) wrote: > Hello, > > We are planning to use ES as a primary datastore. > > Here is my usecase > > We receive a million transactions per day (all are inserts). > Each transaction is around 500KB size, transaction has 10 fields we should > be able to search on all 10 fields. > We want to keep around 1 yr worth of data, this comes around 180TB > > Can you please let me know any problems that might arise if i use elastic > search as the primary datastore. > > Regards, > Suman > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f52b61e2-0955-4e79-8bb8-61c9428c67d1%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f52b61e2-0955-4e79-8bb8-61c9428c67d1%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/f52b61e2-0955-4e79-8bb8-61c9428c67d1%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f52b61e2-0955-4e79-8bb8-61c9428c67d1%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAEM624YHED333JiG8Jb8X8h41HF64xGze-ZoJKrt2R3fxxqn\_A%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAEM624YHED333JiG8Jb8X8h41HF64xGze-ZoJKrt2R3fxxqn_A%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Thomas\_Bolis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/thomas_bolis/32/700_2.png) [@Thomas\_Bolis](https://discuss.elastic.co/u/Thomas_Bolis)\ **Post date:** [September 17, 2014, 12:04pm UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/3 "2014-09-17T12:04:07Z")
Hi, You have to calculate the volumes you will keep in one shard first then you have to break your volumes into the number of shards you will maintain and then scale accordingly into a number of nodes, or at least as your volumes grow you should grow your cluster as well. It is difficult to predict what problems may arise it is too generic your case, what will be the usage of the cluster? what queries you will perform, you will mostly do indexing and occasionally querying or you will intensively query your data. Most important you need to think how you will partition your data, will you have one index, multiple index like a logstash approach? or not Maybe check here: [Sizing Elasticsearch | Elastic Blog](https://www.found.no/foundation/sizing-elasticsearch/) For data more than a year what you will do delete them? Do you afford to lose data? Will you keep backups? IMHO, these are some of the questions you must answer in order to see whether such an approach suit your needs. It is hardware, structure and partitioning of your data. Thomas On Wednesday, 17 September 2014 13:41:55 UTC+3, P Suman wrote: > Hello, > > We are planning to use ES as a primary datastore. > > Here is my usecase > > We receive a million transactions per day (all are inserts). > Each transaction is around 500KB size, transaction has 10 fields we should > be able to search on all 10 fields. > We want to keep around 1 yr worth of data, this comes around 180TB > > Can you please let me know any problems that might arise if i use elastic > search as the primary datastore. > > Regards, > Suman -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alex\_Kamil](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Alex\_Kamil](https://discuss.elastic.co/u/Alex_Kamil)\ **Post date:** [September 17, 2014, 12:48pm UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/4 "2014-09-17T12:48:17Z")
ES is a fantastic search engine but there is some risk [http://aphyr.com/posts/317-call-me-maybe-elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) of data loss, and a few other [https://www.quora.com/Why-should-I-NOT-use-ElasticSearch-as-my-primary-datastore](https://www.quora.com/Why-should-I-NOT-use-ElasticSearch-as-my-primary-datastore) potential disadvantages which might or might not be relevant to you. You can always combine ES via JDBC river [https://github.com/jprante/elasticsearch-river-jdbc](https://github.com/jprante/elasticsearch-river-jdbc) with a stable, secure database, e.g. Mysql [https://www.quora.com/How-do-i-use-Elastic-search-with-mysql-database-I-am-currently-experimenting-with-jdbc-river-but-will-it-be-fast-enough-in-production](https://www.quora.com/How-do-i-use-Elastic-search-with-mysql-database-I-am-currently-experimenting-with-jdbc-river-but-will-it-be-fast-enough-in-production)or Hbase [http://lessc0de.github.io/connecting\_hbase\_to\_elasticsearch.html](http://lessc0de.github.io/connecting_hbase_to_elasticsearch.html), since you have lots of data hbase might be a better option. On Wed, Sep 17, 2014 at 8:04 AM, Thomas [thomas.bolis@gmail.com](mailto:thomas.bolis@gmail.com) wrote: > Hi, > > You have to calculate the volumes you will keep in one shard first then > you have to break your volumes into the number of shards you will maintain > and then scale accordingly into a number of nodes, or at least as your > volumes grow you should grow your cluster as well. > > It is difficult to predict what problems may arise it is too generic your > case, what will be the usage of the cluster? what queries you will perform, > you will mostly do indexing and occasionally querying or you will > intensively query your data. > > Most important you need to think how you will partition your data, will > you have one index, multiple index like a logstash approach? or not > Maybe check here: [Sizing Elasticsearch | Elastic Blog](https://www.found.no/foundation/sizing-elasticsearch/) > > For data more than a year what you will do delete them? Do you afford to > lose data? Will you keep backups? > > IMHO, these are some of the questions you must answer in order to see > whether such an approach suit your needs. It is hardware, structure and > partitioning of your data. > > Thomas > > On Wednesday, 17 September 2014 13:41:55 UTC+3, P Suman wrote: > > > Hello, > > > > We are planning to use ES as a primary datastore. > > > > Here is my usecase > > > > We receive a million transactions per day (all are inserts). > > Each transaction is around 500KB size, transaction has 10 fields we > > should be able to search on all 10 fields. > > We want to keep around 1 yr worth of data, this comes around 180TB > > > > Can you please let me know any problems that might arise if i use elastic > > search as the primary datastore. > > > > Regards, > > Suman > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOtKWX47iRi6P%2BSp-GC%2B8JL1xmwKoL4yHerMC4PG5rYDiL8YXA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOtKWX47iRi6P%2BSp-GC%2B8JL1xmwKoL4yHerMC4PG5rYDiL8YXA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![softwaredoug](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softwaredoug/32/22681_2.png) [@softwaredoug](https://discuss.elastic.co/u/softwaredoug)\ **Post date:** [September 17, 2014, 1:20pm UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/5 "2014-09-17T13:20:55Z")
I'd also suggest checking out DataStax Enterprise -- a commercial flavor of Cassandra. Its Cassandra, so update rates and volume are its strong suit. Its intended as a primary data store. It has a Solr (another search engine) instance on each node that indexes the local data on that node, enabling full text search. Solr is not nearly as user friendly as Elasticsearch, but otherwise there's a lot of comparable features depending on your search needs. ## Doug From: Alex Kamil [alex.kamil@gmail.com](mailto:alex.kamil@gmail.com) Sent: ‎9/‎17/‎2014 8:48 AM To: [elasticsearch@googlegroups.com](mailto:elasticsearch@googlegroups.com) Subject: Re: Using ES as a primary datastore. ES is a fantastic search engine but there is some risk [http://aphyr.com/posts/317-call-me-maybe-elasticsearch](http://aphyr.com/posts/317-call-me-maybe-elasticsearch) of data loss, and a few other [https://www.quora.com/Why-should-I-NOT-use-ElasticSearch-as-my-primary-datastore](https://www.quora.com/Why-should-I-NOT-use-ElasticSearch-as-my-primary-datastore) potential disadvantages which might or might not be relevant to you. You can always combine ES via JDBC river [https://github.com/jprante/elasticsearch-river-jdbc](https://github.com/jprante/elasticsearch-river-jdbc) with a stable, secure database, e.g. Mysql [https://www.quora.com/How-do-i-use-Elastic-search-with-mysql-database-I-am-currently-experimenting-with-jdbc-river-but-will-it-be-fast-enough-in-production](https://www.quora.com/How-do-i-use-Elastic-search-with-mysql-database-I-am-currently-experimenting-with-jdbc-river-but-will-it-be-fast-enough-in-production)or Hbase [http://lessc0de.github.io/connecting\_hbase\_to\_elasticsearch.html](http://lessc0de.github.io/connecting_hbase_to_elasticsearch.html), since you have lots of data hbase might be a better option. On Wed, Sep 17, 2014 at 8:04 AM, Thomas [thomas.bolis@gmail.com](mailto:thomas.bolis@gmail.com) wrote: > Hi, > > You have to calculate the volumes you will keep in one shard first then > you have to break your volumes into the number of shards you will maintain > and then scale accordingly into a number of nodes, or at least as your > volumes grow you should grow your cluster as well. > > It is difficult to predict what problems may arise it is too generic your > case, what will be the usage of the cluster? what queries you will perform, > you will mostly do indexing and occasionally querying or you will > intensively query your data. > > Most important you need to think how you will partition your data, will > you have one index, multiple index like a logstash approach? or not > Maybe check here: [Sizing Elasticsearch | Elastic Blog](https://www.found.no/foundation/sizing-elasticsearch/) > > For data more than a year what you will do delete them? Do you afford to > lose data? Will you keep backups? > > IMHO, these are some of the questions you must answer in order to see > whether such an approach suit your needs. It is hardware, structure and > partitioning of your data. > > Thomas > > On Wednesday, 17 September 2014 13:41:55 UTC+3, P Suman wrote: > > > Hello, > > > > We are planning to use ES as a primary datastore. > > > > Here is my usecase > > > > We receive a million transactions per day (all are inserts). > > Each transaction is around 500KB size, transaction has 10 fields we > > should be able to search on all 10 fields. > > We want to keep around 1 yr worth of data, this comes around 180TB > > > > Can you please let me know any problems that might arise if i use elastic > > search as the primary datastore. > > > > Regards, > > Suman > > > > -- > > You received this message because you are subscribed to the Google Groups > > "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an > > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit > > [https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com) > > [https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0612d5d3-05df-4538-a3f0-e87cd9b3dc49%40googlegroups.com?utm_medium=email&utm_source=footer) > > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOtKWX47iRi6P%2BSp-GC%2B8JL1xmwKoL4yHerMC4PG5rYDiL8YXA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOtKWX47iRi6P%2BSp-GC%2B8JL1xmwKoL4yHerMC4PG5rYDiL8YXA%40mail.gmail.com) [https://groups.google.com/d/msgid/elasticsearch/CAOtKWX47iRi6P%2BSp-GC%2B8JL1xmwKoL4yHerMC4PG5rYDiL8YXA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CAOtKWX47iRi6P%2BSp-GC%2B8JL1xmwKoL4yHerMC4PG5rYDiL8YXA%40mail.gmail.com?utm_medium=email&utm_source=footer) . For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/5188131938126812645%40unknownmsgid](https://groups.google.com/d/msgid/elasticsearch/5188131938126812645%40unknownmsgid). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![elvarb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/elvarb/32/44840_2.png) [@elvarb](https://discuss.elastic.co/u/elvarb)\ **Post date:** [November 24, 2014, 7:25pm UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/6 "2014-11-24T19:25:46Z")
Like others have mentioned, there is a risk of data loss at the moment but Elasticsearch is working on making it better and better. I watched this video about Couchbase and Elasticsearch, [https://www.youtube.com/watch?v=rpwtxpmuDb0](https://www.youtube.com/watch?v=rpwtxpmuDb0) After that then I highly recommend researching Couchbase as an option. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/bff701c2-7885-4e8b-8ddd-968172ab4492%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bff701c2-7885-4e8b-8ddd-968172ab4492%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Alex\_Kamil](https://avatars.discourse-cdn.com/v4/letter/a/d9b06d/32.png) [@Alex\_Kamil](https://discuss.elastic.co/u/Alex_Kamil)\ **Post date:** [November 24, 2014, 8:43pm UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/7 "2014-11-24T20:43:26Z")
in the spirit of this thread let me plug my favorite database here: Connecting Hbase to Elasticsearch in 10 min or less [http://lessc0de.github.io/connecting\_hbase\_to\_elasticsearch.html](http://lessc0de.github.io/connecting_hbase_to_elasticsearch.html) On Mon, Nov 24, 2014 at 2:25 PM, Elvar Böðvarsson [elvarb@gmail.com](mailto:elvarb@gmail.com) wrote: > Like others have mentioned, there is a risk of data loss at the moment but > Elasticsearch is working on making it better and better. > > I watched this video about Couchbase and Elasticsearch, > [https://www.youtube.com/watch?v=rpwtxpmuDb0](https://www.youtube.com/watch?v=rpwtxpmuDb0) > > After that then I highly recommend researching Couchbase as an option. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/bff701c2-7885-4e8b-8ddd-968172ab4492%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/bff701c2-7885-4e8b-8ddd-968172ab4492%40googlegroups.com) > [https://groups.google.com/d/msgid/elasticsearch/bff701c2-7885-4e8b-8ddd-968172ab4492%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/bff701c2-7885-4e8b-8ddd-968172ab4492%40googlegroups.com?utm_medium=email&utm_source=footer) > . > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAOtKWX6%3DiZ2pd7NEzYVM2vXhVJ1KpeoQy5oYh-qbVC6G6KTVEw%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAOtKWX6%3DiZ2pd7NEzYVM2vXhVJ1KpeoQy5oYh-qbVC6G6KTVEw%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:47am UTC](https://discuss.elastic.co/t/using-es-as-a-primary-datastore/19839/8 "2017-07-06T00:47:59Z")