Elasticsearch hardware requirement

Elastic Stack Elasticsearch
1

Hi list,

I currently have a 4 nodes cluster to collect varnish logs of my platform,
The cluster is dealing with almost 80Mill of documents (2000 documents per
second) and sometime I have some performance problems.
Let me explain how i do it. In all frontend server i have a logstash agent
sending Varnish logs to a redis instance in one of my elasticsearch
machines. In that machine other logstash process pull the events and store
it in the elasticsearch cluster. My problem come when the logs begin to
queue in redis because elasticsearch cluster is not able to store so fast.

The cluster machines (virtual machines) have 4 CPU and 4GB of RAM and I
think are not enough to deal with 2000-3000 events per second. I have been
reading a lot about this issue and everybody have machines bigger than
mine. What would be the right size for the machines? I have an index per
day which size is almost 100GB, and I've read that the size of your RAM
should be the same that your indexes size, but that is not possible for me.
If I add more nodes to the cluster the performance will be better?

Thanks in advance!

Jorge

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d21ce426-7612-47a1-9da0-56441d544892%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

The best way is to test it. Take 1 node with say 16GB of RAM and allocate
8GB to ES. Then start pushing 1 day worth of logs into an index with 5
shards and 0 replicas on that one node AND run your typical queries. Take
measurements like throughput/dps, query latency, ram usage, cpu, and disk.
Then you'll know how much you can do on a single node and then start
extrapolating from there.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9ee91a2d-2941-4327-82f0-2793eb1cb242%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

Thanks for the reply. I have done the test with 1 node (16GB RAM and 8CPUs,
allocating 8GB to ES), and I have been able to deal with all events with
only 1 node. Now i¹m trying to find out where is the bottleneck.

Next step, I¹m gonna try to benchmarking elasticsearch without external
elements in order to take measures.

Best Regards!

Jorge Román Novalbos
CEO
jroman@servotic.com
679 99 08 62
http://www.linkedin.com/in/jorgeromanwebperformance
https://twitter.com/servoticsl http://www.facebook.com/servotic
skype:jorgeroman1980?call
http://www.servotic.com

Este mensaje es solamente para la persona a la que va dirigido. Puede
contener información confidencial o legalmente protegida. Si usted ha
recibido este mensaje por error, le rogamos que borre de su sistema el
mensaje inmediatamente y notifíquelo al remitente. No debe, directa o
indirectamente, usar, revelar, distribuir, imprimir o copiar ninguna de las
partes de este mensaje si no es usted el destinatario. En cumplimiento de la
Ley Orgánica 15/1999, de Protección de Datos de Carácter Personal le
informamos que su dirección de correo electrónico, sus datos personales y de
empresa pasarán a formar parte de nuestro fichero de Clientes y Proveedores,
registrado ante la Agencia de Protección de Datos. En cumplimiento de la Ley
34/2002, de Servicios de la Sociedad de la Información y el Comercio
Electrónico, le informamos que esta dirección de correo electrónico podrá
ser utilizada para el envío de información comercial o promocional de
nuestra organización. Si no desea recibir información o desea ejercitar sus
derechos de acceso, rectificación, cancelación y oposición, le rogamos nos
lo comunique vía correo electrónico a la siguiente dirección:
lopd@servotic.com

De: Binh Ly binhly_es@yahoo.com
Responder a: elasticsearch@googlegroups.com
Fecha: Fri, 28 Mar 2014 07:06:18 -0700 (PDT)
Para: elasticsearch@googlegroups.com
Asunto: Re: Elasticsearch hardware requirement

The best way is to test it. Take 1 node with say 16GB of RAM and allocate
8GB to ES. Then start pushing 1 day worth of logs into an index with 5
shards and 0 replicas on that one node AND run your typical queries. Take
measurements like throughput/dps, query latency, ram usage, cpu, and disk.
Then you'll know how much you can do on a single node and then start
extrapolating from there.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/tVywigD5iU8/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/9ee91a2d-2941-4327-82f0-2793
eb1cb242%40googlegroups.com
<https://groups.google.com/d/msgid/elasticsearch/9ee91a2d-2941-4327-82f0-279
3eb1cb242%40googlegroups.com?utm_medium=email&utm_source=footer> .
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CF619208.1737D%jroman%40servotic.com.
For more options, visit https://groups.google.com/d/optout.