# 2 TB of logs per day in ES? **URL:** \ **Category:** Elasticsearch\ **Created:** [October 9, 2013, 9:33pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894 "2013-10-09T21:33:59Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![Alexios\_Giotis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexios_giotis/32/2060_2.png) [@Alexios\_Giotis](https://discuss.elastic.co/u/Alexios_Giotis)\ **Post date:** [October 9, 2013, 9:33pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/1 "2013-10-09T21:33:59Z")
Hi, After reading > **[Using ElasticSearch and Logstash to Serve Billions of Searchable Events for...](https://www.elastic.co/blog/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers)** > > We're pleased to welcome our first guest blogger, Ralph Meijer, Software Developer at Rackspace working on Mailgun. Ralph graciously offered to send us a write up of Mailgun's use of Elasticsearch aft... I am wondering if ES can handle the bigger volume of a potential customer. This is: - 2 TB of input logs per day - 95000 events per second sustained, 120000 events per second peak - 30 days retention - Response time up to 2 seconds. Besides ES, we are planning to use logstash, kibana 3, redis and some proprietary extensions. If ES can handle it, any estimate of the hardware needed ? (# of nodes, vCPUs, RAM & storage per node) If this is a large volume for ES, any other suggestion ? Thank you, Alex Giotis -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 9, 2013, 9:47pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/2 "2013-10-09T21:47:51Z")
Sure, ES can handle this. How many queries per second? Set up a single node first, check the events per second you can index (and query), then a second node, then x2 for replication etc and estimate the scaling factor for 120,000 events per second. You get the idea. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Bill\_de\_hOra](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bill_de_hora/32/2059_2.png) [@Bill\_de\_hOra](https://discuss.elastic.co/u/Bill_de_hOra)\ **Post date:** [October 9, 2013, 9:59pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/3 "2013-10-09T21:59:38Z")
> Sure, ES can handle this. Can you point at a system/paper to back that up? Knowing operationally how to to replay data or deal with cluster failures under this kind of load would be very useful. Bill On 2013/10/09 22:47, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > Sure, ES can handle this. > > How many queries per second? > > Set up a single node first, check the events per second you can index > (and query), then a second node, then x2 for replication etc and > estimate the scaling factor for 120,000 events per second. You get the idea. > > Jörg > > -- > You received this message because you are subscribed to the Google > Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send > an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Norberto\_Meijome](https://avatars.discourse-cdn.com/v4/letter/n/ed655f/32.png) [@Norberto\_Meijome](https://discuss.elastic.co/u/Norberto_Meijome)\ **Post date:** [October 9, 2013, 10:24pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/4 "2013-10-09T22:24:54Z")
You may find logstash being another bottleneck ... and somewhat more unpredictable than ES in this regard...YMMV. On 10/10/2013 8:34 AM, "Alexios Giotis" [alex.giotis@gmail.com](mailto:alex.giotis@gmail.com) wrote: > Hi, > > After reading > [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/blog/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers/I) am wondering if ES can handle the bigger volume of a potential customer. > This is: > > - 2 TB of input logs per day > - 95000 events per second sustained, 120000 events per second peak > - 30 days retention > - Response time up to 2 seconds. > > Besides ES, we are planning to use logstash, kibana 3, redis and some > proprietary extensions. > > If ES can handle it, any estimate of the hardware needed ? (# of nodes, > vCPUs, RAM & storage per node) > If this is a large volume for ES, any other suggestion ? > > Thank you, > Alex Giotis > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![Alexios\_Giotis](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alexios_giotis/32/2060_2.png) [@Alexios\_Giotis](https://discuss.elastic.co/u/Alexios_Giotis)\ **Post date:** [October 9, 2013, 10:32pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/5 "2013-10-09T22:32:56Z")
Jörg, Thank you for the reply. A couple of administrators will monitor the data using kibana, so I guess there will be 1 or 2 queries per second. Tomorrow, I will try to generate some data for testing with a couple of systems internally or a few more on Amazon EC2. But it will be challenging and expensive to feed an ES cluster with 60 TB of data and I would like any related hint from people with a similar volume. For example, what is an estimate of the required storage ? 60TB x 2 (replication) x 10 (indexing) = 1.2 PB ? Alex > How many queries per second? > > Set up a single node first, check the events per second you can index (and > query), then a second node, then x2 for replication etc and estimate the > scaling factor for 120,000 events per second. You get the idea. > > Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\ **Post date:** [October 9, 2013, 10:32pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/6 "2013-10-09T22:32:59Z")
We have around 13TB (40TB replicated) and we recover a node/cluster restart within approx two hours. That's on 10Gb ethernet utilising cluster.routing.allocation.disable\_allocation, and running across 7 data nodes. The last time a node actually failed it took around 6 hours to recover, however I think we could reduce that with some tweaking. Regards, Mark Walkom Infrastructure Engineer Campaign Monitor email: [markw@campaignmonitor.com](mailto:markw@campaignmonitor.com) web: [www.campaignmonitor.com](http://www.campaignmonitor.com) On 10 October 2013 08:59, Bill de hÓra [bill@dehora.net](mailto:bill@dehora.net) wrote: > > Sure, ES can handle this. > > Can you point at a system/paper to back that up? Knowing operationally how > to to replay data or deal with cluster failures under this kind of load > would be very useful. > > Bill > > On 2013/10/09 22:47, [joergprante@gmail.com](mailto:joergprante@gmail.com) wrote: > > > Sure, ES can handle this. > > > > How many queries per second? > > > > Set up a single node first, check the events per second you can index > > (and query), then a second node, then x2 for replication etc and > > estimate the scaling factor for 120,000 events per second. You get the > > idea. > > > > Jörg > > > > -- > > You received this message because you are subscribed to the Google > > Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send > > an email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > > . > > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > > . > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to elasticsearch+unsubscribe@\*\*[googlegroups.com](http://googlegroups.com)[elasticsearch%2Bunsubscribe@googlegroups.com](mailto:elasticsearch%2Bunsubscribe@googlegroups.com) > . > For more options, visit [https://groups.google.com/\*\*groups/opt\_out](https://groups.google.com/**groups/opt_out)[https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out) > . -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 9, 2013, 10:49pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/7 "2013-10-09T22:49:44Z")
Github's Elasticsearch team has some experience with 30TB+ primary data, and I like how they share their learning curve in this interview [http://exploringelasticsearch.com/book/elasticsearch-at-scale-interviews/interview-with-the-github-elasticsearch-team.html](http://exploringelasticsearch.com/book/elasticsearch-at-scale-interviews/interview-with-the-github-elasticsearch-team.html) Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [October 9, 2013, 11:10pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/8 "2013-10-09T23:10:22Z")
Not sure how you get this formula. You should at most allocate ~50% disk space on each node to have some space reserved for temporary segments and additional shards moving around (recovery). Then you need to estimate the compression factor, because ES compresses data by default. This can be a factor of roughly 4 or 5 for usual log files. Example: 2TB disk size and 4 disks in a RAID0 at each machine, there are 4TB index size per node available, which gives 4 nodes having 16TB, after indexing ~64TB input. The more nodes the better. I doubt that 4 nodes can handle 120,000 events per second. Let's assume 10,000 events can be handled by a node. So you may need 12 nodes, just working 100% for indexing. And between Amazon EC2 and bare metal servers there is quite a difference, which is also a factor in the formula. Jörg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out). ---
**Author:** ![ananth](https://avatars.discourse-cdn.com/v4/letter/a/dfb087/32.png) [@ananth](https://discuss.elastic.co/u/ananth)\ **Post date:** [September 29, 2014, 12:12pm UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/9 "2014-09-29T12:12:13Z")
Hi Alexios, Hope you have scaled 2TB per day. We are almost in same situation . We are getting 1 TB of data per day / 30 days retention. Can you share your hardware details (# of nodes, vCPUs, RAM & storage per node) ? Please share your experience , this will be helpful to us and save our time. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/6bb43f90-b6b7-494d-a4ea-2789e9c549fe%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/6bb43f90-b6b7-494d-a4ea-2789e9c549fe%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 12:59am UTC](https://discuss.elastic.co/t/2-tb-of-logs-per-day-in-es/13894/10 "2017-07-06T00:59:20Z")