2 TB of logs per day in ES?

Hi,

After reading


I am wondering if ES can handle the bigger volume of a potential customer.
This is:

  • 2 TB of input logs per day
  • 95000 events per second sustained, 120000 events per second peak
  • 30 days retention
  • Response time up to 2 seconds.

Besides ES, we are planning to use logstash, kibana 3, redis and some
proprietary extensions.

If ES can handle it, any estimate of the hardware needed ? (# of nodes,
vCPUs, RAM & storage per node)
If this is a large volume for ES, any other suggestion ?

Thank you,
Alex Giotis

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sure, ES can handle this.

How many queries per second?

Set up a single node first, check the events per second you can index (and
query), then a second node, then x2 for replication etc and estimate the
scaling factor for 120,000 events per second. You get the idea.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Sure, ES can handle this.

Can you point at a system/paper to back that up? Knowing operationally
how to to replay data or deal with cluster failures under this kind of
load would be very useful.

Bill

On 2013/10/09 22:47, joergprante@gmail.com wrote:

Sure, ES can handle this.

How many queries per second?

Set up a single node first, check the events per second you can index
(and query), then a second node, then x2 for replication etc and
estimate the scaling factor for 120,000 events per second. You get the idea.

Jörg

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

You may find logstash being another bottleneck ... and somewhat more
unpredictable than ES in this regard...YMMV.
On 10/10/2013 8:34 AM, "Alexios Giotis" alex.giotis@gmail.com wrote:

Hi,

After reading
http://www.elasticsearch.org/blog/using-elasticsearch-and-logstash-to-serve-billions-of-searchable-events-for-customers/I am wondering if ES can handle the bigger volume of a potential customer.
This is:

  • 2 TB of input logs per day
  • 95000 events per second sustained, 120000 events per second peak
  • 30 days retention
  • Response time up to 2 seconds.

Besides ES, we are planning to use logstash, kibana 3, redis and some
proprietary extensions.

If ES can handle it, any estimate of the hardware needed ? (# of nodes,
vCPUs, RAM & storage per node)
If this is a large volume for ES, any other suggestion ?

Thank you,
Alex Giotis

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Jörg,
Thank you for the reply. A couple of administrators will monitor the data
using kibana, so I guess there will be 1 or 2 queries per second.

Tomorrow, I will try to generate some data for testing with a couple of
systems internally or a few more on Amazon EC2. But it will be challenging
and expensive to feed an ES cluster with 60 TB of data and I would like any
related hint from people with a similar volume. For example, what is an
estimate of the required storage ? 60TB x 2 (replication) x 10 (indexing) =
1.2 PB ?

Alex

How many queries per second?

Set up a single node first, check the events per second you can index (and
query), then a second node, then x2 for replication etc and estimate the
scaling factor for 120,000 events per second. You get the idea.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

We have around 13TB (40TB replicated) and we recover a node/cluster restart
within approx two hours.
That's on 10Gb ethernet utilising
cluster.routing.allocation.disable_allocation, and running across 7 data
nodes.

The last time a node actually failed it took around 6 hours to recover,
however I think we could reduce that with some tweaking.

Regards,
Mark Walkom

Infrastructure Engineer
Campaign Monitor
email: markw@campaignmonitor.com
web: www.campaignmonitor.com

On 10 October 2013 08:59, Bill de hÓra bill@dehora.net wrote:

Sure, ES can handle this.

Can you point at a system/paper to back that up? Knowing operationally how
to to replay data or deal with cluster failures under this kind of load
would be very useful.

Bill

On 2013/10/09 22:47, joergprante@gmail.com wrote:

Sure, ES can handle this.

How many queries per second?

Set up a single node first, check the events per second you can index
(and query), then a second node, then x2 for replication etc and
estimate the scaling factor for 120,000 events per second. You get the
idea.

Jörg

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@**googlegroups.comelasticsearch%2Bunsubscribe@googlegroups.com
.
For more options, visit https://groups.google.com/**groups/opt_outhttps://groups.google.com/groups/opt_out
.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Github's Elasticsearch team has some experience with 30TB+ primary data,
and I like how they share their learning curve in this interview

http://exploringelasticsearch.com/book/elasticsearch-at-scale-interviews/interview-with-the-github-elasticsearch-team.html

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Not sure how you get this formula. You should at most allocate ~50% disk
space on each node to have some space reserved for temporary segments and
additional shards moving around (recovery). Then you need to estimate the
compression factor, because ES compresses data by default. This can be a
factor of roughly 4 or 5 for usual log files. Example: 2TB disk size and 4
disks in a RAID0 at each machine, there are 4TB index size per node
available, which gives 4 nodes having 16TB, after indexing ~64TB input. The
more nodes the better. I doubt that 4 nodes can handle 120,000 events per
second. Let's assume 10,000 events can be handled by a node. So you may
need 12 nodes, just working 100% for indexing. And between Amazon EC2 and
bare metal servers there is quite a difference, which is also a factor in
the formula.

Jörg

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Alexios,

Hope you have scaled 2TB per day. We are almost in same situation . We are
getting 1 TB of data per day / 30 days retention.

Can you share your hardware details (# of nodes, vCPUs, RAM & storage per
node) ?

Please share your experience , this will be helpful to us and save our time.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/6bb43f90-b6b7-494d-a4ea-2789e9c549fe%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.