I am wondering if ES can handle the bigger volume of a potential customer.
This is:
2 TB of input logs per day
95000 events per second sustained, 120000 events per second peak
30 days retention
Response time up to 2 seconds.
Besides ES, we are planning to use logstash, kibana 3, redis and some
proprietary extensions.
If ES can handle it, any estimate of the hardware needed ? (# of nodes,
vCPUs, RAM & storage per node)
If this is a large volume for ES, any other suggestion ?
Set up a single node first, check the events per second you can index (and
query), then a second node, then x2 for replication etc and estimate the
scaling factor for 120,000 events per second. You get the idea.
Can you point at a system/paper to back that up? Knowing operationally
how to to replay data or deal with cluster failures under this kind of
load would be very useful.
Set up a single node first, check the events per second you can index
(and query), then a second node, then x2 for replication etc and
estimate the scaling factor for 120,000 events per second. You get the idea.
You may find logstash being another bottleneck ... and somewhat more
unpredictable than ES in this regard...YMMV.
On 10/10/2013 8:34 AM, "Alexios Giotis" alex.giotis@gmail.com wrote:
95000 events per second sustained, 120000 events per second peak
30 days retention
Response time up to 2 seconds.
Besides ES, we are planning to use logstash, kibana 3, redis and some
proprietary extensions.
If ES can handle it, any estimate of the hardware needed ? (# of nodes,
vCPUs, RAM & storage per node)
If this is a large volume for ES, any other suggestion ?
Jörg,
Thank you for the reply. A couple of administrators will monitor the data
using kibana, so I guess there will be 1 or 2 queries per second.
Tomorrow, I will try to generate some data for testing with a couple of
systems internally or a few more on Amazon EC2. But it will be challenging
and expensive to feed an ES cluster with 60 TB of data and I would like any
related hint from people with a similar volume. For example, what is an
estimate of the required storage ? 60TB x 2 (replication) x 10 (indexing) =
1.2 PB ?
Alex
How many queries per second?
Set up a single node first, check the events per second you can index (and
query), then a second node, then x2 for replication etc and estimate the
scaling factor for 120,000 events per second. You get the idea.
We have around 13TB (40TB replicated) and we recover a node/cluster restart
within approx two hours.
That's on 10Gb ethernet utilising
cluster.routing.allocation.disable_allocation, and running across 7 data
nodes.
The last time a node actually failed it took around 6 hours to recover,
however I think we could reduce that with some tweaking.
On 10 October 2013 08:59, Bill de hÓra bill@dehora.net wrote:
Sure, ES can handle this.
Can you point at a system/paper to back that up? Knowing operationally how
to to replay data or deal with cluster failures under this kind of load
would be very useful.
Set up a single node first, check the events per second you can index
(and query), then a second node, then x2 for replication etc and
estimate the scaling factor for 120,000 events per second. You get the
idea.
Not sure how you get this formula. You should at most allocate ~50% disk
space on each node to have some space reserved for temporary segments and
additional shards moving around (recovery). Then you need to estimate the
compression factor, because ES compresses data by default. This can be a
factor of roughly 4 or 5 for usual log files. Example: 2TB disk size and 4
disks in a RAID0 at each machine, there are 4TB index size per node
available, which gives 4 nodes having 16TB, after indexing ~64TB input. The
more nodes the better. I doubt that 4 nodes can handle 120,000 events per
second. Let's assume 10,000 events can be handled by a node. So you may
need 12 nodes, just working 100% for indexing. And between Amazon EC2 and
bare metal servers there is quite a difference, which is also a factor in
the formula.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.