# High Scale Use Case **URL:** \ **Category:** Elasticsearch\ **Created:** [December 20, 2016, 12:53pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569 "2016-12-20T12:53:12Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![SGE](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@SGE](https://discuss.elastic.co/u/SGE)\ **Post date:** [December 20, 2016, 12:53pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569/1 "2016-12-20T12:53:12Z")
Hi, Our system currently uses other NoSQL data store to manage its data. It currently supports 180,000 inserts per second (we never update), each document is 1KB and is stored in the system for 180 days yielding 2.6PB of raw data. The system continuously analyzes the data and allows the users to browse analysis resultss and to randomely access the stored documents (by id). We have a new requirement to support advanced search capabilities over a subset of fields (20 out of 280). We are considering using Elasticsearch in one of two ways: 1. As a search engine only (with \_source disabled) - we index the 20 fields and when the user performs a search we retrieve the ids from elasticsearch and then use them to query the actual documents from a document store (e.g. - cassandre). 2. As both search engine and document store (with \_source enabled) What would be the better approach considering the high insert rate ? Thanks ---
**Author:** ![jpountz](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jpountz/32/45836_2.png) [@jpountz](https://discuss.elastic.co/u/jpountz)\ **Post date:** [December 20, 2016, 1:28pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569/2 "2016-12-20T13:28:23Z")
When it comes to indexing speed, `_source` in not the bottleneck in my experience: it is cheap to add to the index as it is mostly about compressing chunks of 16KB and writing them sequentially. The bottleneck is usually more about analyzing documents (computing tokens) and indexing them, which involves a lot of sorting for instance. Maybe you should rather look into the complexity of your indexing chain, ie. which analyzers you are using, how many fields you have, etc. `_source` will have to be evaluated too, but it should come later in the decision process I think. To me the biggest benefit of disabling `_source` is to make the index smaller to improve how much of the index the file system cache can cover. ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 20, 2016, 1:29pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569/3 "2016-12-20T13:29:11Z")
Elasticsearch is more designed around the second use case. Aggregations are all implemented against a column store built when the documents are indexed. Fetching IDs is not implemented against the column store. It uses the more compact storage mechanism used to store \_source which works fine if you want to fetch a few of them, but poorly for millions. Fetching all the results requires this `scroll` mechanism to let you get deep into the results set and it has more overhead than elasticsearch doing the aggregation on its own. A couple things: - Aggregations have fairly limited support for join-like concepts, mostly parent/child like stuff. If you are used to arbitrary joins you are likely going to miss something. I'd index a few million docs and test. - You don't have to store the \_source to do the aggregations. It is worth investigating not storing it if you have another spot for it. But if you are thinking of shifting the hardware you were using then maybe you do need to store it so you can look at the document. - Looks at the shrink/rollover APIs to help manage your indexes. Experiment with index sizes and such. This is enough data that you'll have to find some comfortable index sizes to work with. - Looks at hot/warm machines and migrating data from one to the other. It might be a good setup if you don't usually query all the data. Like if you usually query the first month and only sometimes go back all 180 days. - Make sure you clean up the data that rolls off the end of your time window by deleting the index it is in. Deleting documents doesn't work as well as deleting the whole index. - Beware elasticsearch's dynamic mapping updates. They work fine for demonstrations but have scalability issues with lots of documents at once. Also having a zillion different fields doesn't work super well. The column store isn't supper happy with sparse values. It is OK, but not great. Folks run elasticsearch at this scale, but it is big enough that you'll need to pay some attention to it to get it humming along. ---
**Author:** ![SGE](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@SGE](https://discuss.elastic.co/u/SGE)\ **Post date:** [December 20, 2016, 7:26pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569/4 "2016-12-20T19:26:53Z")
Thanks for the quick and throrogh replies. I have two more questions: 1- According to my calculations, the first approach (\_source enabled) will result in roughly 200TB while the second approach will result in 2700TB being managed by ES. Do these extra 2500TB have any implication other than additional storage requirements? (I've read something about increasing the likelyhood of index corruption taking place) 2- Is it best practice to use ES as a document store which is its own source-of-truth? ---
**Author:** ![nik9000](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/nik9000/32/44947_2.png) [@nik9000](https://discuss.elastic.co/u/nik9000)\ **Post date:** [December 20, 2016, 8:08pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569/5 "2016-12-20T20:08:11Z")
For the first question it is probably worth experimenting with. It'll certainly take longer to replicate those indexes. If you don't need the source documents you certainly can turn off storing them. Source usually doesn't cause as drastic a size change as you've calculated though so I'm not sure what is up there. As far as source of truth-ness. [This](https://www.elastic.co/guide/en/elasticsearch/resiliency/current/index.html) is a good read. It really depends on your tolerance. Elasticsearch has a ways to go to before I'd consider it rock solid but for many uses it is fine. ---
**Author:** ![SGE](https://avatars.discourse-cdn.com/v4/letter/s/67e7ee/32.png) [@SGE](https://discuss.elastic.co/u/SGE)\ **Post date:** [December 21, 2016, 12:38pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569/6 "2016-12-21T12:38:38Z")
Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [January 18, 2017, 12:39pm UTC](https://discuss.elastic.co/t/high-scale-use-case/69569/7 "2017-01-18T12:39:08Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.