Goldman Sachs gave a talk about how they're using Elasticsearch to index
5TB of log data per day. I can't find the video of the talk, but from a
blogpost about it:
Next was Indy Tharmakumar from our hosts Goldman
Sachshttp://www.goldmansachs.com/,
showing how his team have built powerful support systems using
Elasticsearch to index log data. Using 32 1 core CPU instances the system
they have built can store 1.2 billion log lines with a throughput up to
40,000 messages a second (the systems monitored produce 5TB of log data
every day). Log data is queued up in Redis http://redis.io/, distributed
to many Logstash http://logstash.net/ processes, indexed by Elasticsearch
with a Kibana http://rashidkpc.github.io/Kibana/ front end. They learned
that Logstash can be particularly CPU intensive but Elasticsearch itself
scales extremely well. Future plans include considering Apache
Kafkahttp://kafka.apache.org/ as
a data backbone.
On 19 April 2014 06:46, 熊贻青 xiong.jaguar@gmail.com wrote:
We have a cluster with 10 nodes, 48g heap for each ES process. The total
indexing rate is about 25000 doc per second, about 20 indices actively
receiving new data. I'm really courious to compare and evaluate the
indexing performance numers.Thanks!
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAP0hgQ34ZwY6Or0PUFZn_Ciu_iyZZJjyXfz%3DNBu64Ge9uN3hxQ%40mail.gmail.comhttps://groups.google.com/d/msgid/elasticsearch/CAP0hgQ34ZwY6Or0PUFZn_Ciu_iyZZJjyXfz%3DNBu64Ge9uN3hxQ%40mail.gmail.com?utm_medium=email&utm_source=footer
.For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKS6AnOVKwNszU-SFYmyGUpk57U_kz8iXQssGk%3DX81KMiA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.