# How about elasticsearch for a heavy size load? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 4, 2012, 4:05am UTC](https://discuss.elastic.co/t/how-about-elasticsearch-for-a-heavy-size-load/9236 "2012-10-04T04:05:34Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![Hernan\_Leoni](https://avatars.discourse-cdn.com/v4/letter/h/6f9a4e/32.png) [@Hernan\_Leoni](https://discuss.elastic.co/u/Hernan_Leoni)\ **Post date:** [October 4, 2012, 4:05am UTC](https://discuss.elastic.co/t/how-about-elasticsearch-for-a-heavy-size-load/9236/1 "2012-10-04T04:05:34Z")
Hi group, I will be starting some testing using elastic search for a really heavy amount of data (at least heavy for me by now) I will need to handle a cluster handling about 5000GB of text for searching Should it be better to have different smaller clusters, organized depending on my search needs? Should I handle scheduled indexing in some way? I would need searches to finish in some acceptable time for an end user performing queries, but I don't expect to have a heavy load on queries, But well, I'm worried about the size of that data, it will millions of small entires, I guess each entry would be less than 1KB, with an average around 200 bytes. I would thank so much some comments about some experiences like this. Thanks in advance, Hernán -- ---
**Author:** ![Radim](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/radim/32/2213_2.png) [@Radim](https://discuss.elastic.co/u/Radim)\ **Post date:** [October 4, 2012, 6:39am UTC](https://discuss.elastic.co/t/how-about-elasticsearch-for-a-heavy-size-load/9236/2 "2012-10-04T06:39:34Z")
Hello Hernán, plain indexing is not a problem as such. The challenge is making queries performant over your index. And advice here will differ depending on your queries: will you be using facets? Will your queries only hit a subset of all data (such as period-based queries)? How often will you be indexing new documents? In batches? Depending on that, you can optimize the structure of your indexes, the number of shards, refresh interval etc. See kimchy's recent talk about different scenarios: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/videos/2012/06/05/big-data-search-and-analytics.html) Also be aware of compression options: > **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)** > > Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today. Best, Radim On Oct 4, 6:05 am, Hernán Leoni [leoni.her...@gmail.com](mailto:leoni.her...@gmail.com) wrote: > Hi group, > > I will be starting some testing using Elasticsearch for a really heavy > amount of data (at least heavy for me by now) > I will need to handle a cluster handling about 5000GB of text for searching > Should it be better to have different smaller clusters, organized depending > on my search needs? > Should I handle scheduled indexing in some way? > I would need searches to finish in some acceptable time for an end user > performing queries, but I don't expect to have a heavy load on queries, > But well, I'm worried about the size of that data, it will millions of > small entires, I guess each entry would be less than 1KB, with an average > around 200 bytes. > I would thank so much some comments about some experiences like this. > > Thanks in advance, > > Hernán -- ---
**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\ **Post date:** [October 4, 2012, 3:10pm UTC](https://discuss.elastic.co/t/how-about-elasticsearch-for-a-heavy-size-load/9236/3 "2012-10-04T15:10:19Z")
Hi Hernán, It reeeeally depends. 🙂 5 TB is not small, but doable, depending on a number of factors: hardware, ES configuration, query complexity, query concurrency, query latency requirements, etc. Unfortunately, nobody can give you precise advice without knowing a lot more details about the above.... you'll want to look at sharding, oversharding, replication, at cache sizes, at compression, at routing and filtering, etc. etc. Again, can't give you exact guidance or answers without knowing a lot more. ## Otis Search Analytics - [Cloud Monitoring Tools & Services | Sematext](http://sematext.com/search-analytics/index.html) Performance Monitoring - [Sematext Monitoring | Infrastructure Monitoring Service](http://sematext.com/spm/index.html) On Thursday, October 4, 2012 12:05:40 AM UTC-4, Hernán Leoni wrote: > Hi group, > > I will be starting some testing using Elasticsearch for a really heavy > amount of data (at least heavy for me by now) > I will need to handle a cluster handling about 5000GB of text for searching > Should it be better to have different smaller clusters, organized > depending on my search needs? > Should I handle scheduled indexing in some way? > I would need searches to finish in some acceptable time for an end user > performing queries, but I don't expect to have a heavy load on queries, > But well, I'm worried about the size of that data, it will millions of > small entires, I guess each entry would be less than 1KB, with an average > around 200 bytes. > I would thank so much some comments about some experiences like this. > > Thanks in advance, > > Hernán -- ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:10am UTC](https://discuss.elastic.co/t/how-about-elasticsearch-for-a-heavy-size-load/9236/4 "2017-07-06T03:10:07Z")