Elasticsearch index plan

Dear all,
I am trying index 7000 messages per second which each message is 1kbyte. No
analyzer and no ttl. I have a machine with 2.0G 24 core cup, 380g memory,
and 300g sas *8. Is this machine powerful enough for this?

Thanks and happy holiday!

Alan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWKJxL4kC_ScZrBqfNskECiq8bqRdSnJunsqxpf%2BjcSOg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

It depends on the power of the indexer program you want to use, you should
benchmark this for yourself.

If you index valuable data, one machine is not enough for fault tolerance.

Jörg

On Thu, Dec 25, 2014 at 3:26 PM, Wang cnwangyong@gmail.com wrote:

Dear all,
I am trying index 7000 messages per second which each message is 1kbyte.
No analyzer and no ttl. I have a machine with 2.0G 24 core cup, 380g
memory, and 300g sas *8. Is this machine powerful enough for this?

Thanks and happy holiday!

Alan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWKJxL4kC_ScZrBqfNskECiq8bqRdSnJunsqxpf%2BjcSOg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWKJxL4kC_ScZrBqfNskECiq8bqRdSnJunsqxpf%2BjcSOg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEAGY%2BtPM18np3boPy1cZjUH%3DH_tohK-53Ubj50Y%2BzFWA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thank you Joerg, in my case, all logs will be indexed by flume through
elasticsearch sink, so, only if flume is power enough to feed 7000eps, es
running on this machine is also power enough to accept?

I am also planning a cluster with 3 node, all of them are configured
as master+data node
And the minimum master node is set to 2. Can I configure my elasticsearch
sinks index to multiple nodes simultaneously? Is this a good idea ti
increase index performance?

Thanks,

Alan

On Thursday, December 25, 2014, joergprante@gmail.com joergprante@gmail.com
wrote:

It depends on the power of the indexer program you want to use, you should
benchmark this for yourself.

If you index valuable data, one machine is not enough for fault tolerance.

Jörg

On Thu, Dec 25, 2014 at 3:26 PM, Wang <cnwangyong@gmail.com
<javascript:_e(%7B%7D,'cvml','cnwangyong@gmail.com');>> wrote:

Dear all,
I am trying index 7000 messages per second which each message is 1kbyte.
No analyzer and no ttl. I have a machine with 2.0G 24 core cup, 380g
memory, and 300g sas *8. Is this machine powerful enough for this?

Thanks and happy holiday!

Alan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','elasticsearch%2Bunsubscribe@googlegroups.com');>
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWKJxL4kC_ScZrBqfNskECiq8bqRdSnJunsqxpf%2BjcSOg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWKJxL4kC_ScZrBqfNskECiq8bqRdSnJunsqxpf%2BjcSOg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com
<javascript:_e(%7B%7D,'cvml','elasticsearch%2Bunsubscribe@googlegroups.com');>
.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEAGY%2BtPM18np3boPy1cZjUH%3DH_tohK-53Ubj50Y%2BzFWA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEAGY%2BtPM18np3boPy1cZjUH%3DH_tohK-53Ubj50Y%2BzFWA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWq%3DX75DE3uA7uaEbohey8fQtGH%2BW3hsv_eT3U9%3DsW_MA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

It is the default configuration that Elasticsearch uses all nodes, if you
configure the shard number of an index to the number of nodes you have.

I am not familiar with Flume but Elasticsearch is not the bottleneck if
configured carefully. You must check your I/O subsystem for the write load.

Jörg

On Sat, Dec 27, 2014 at 12:55 PM, Wang cnwangyong@gmail.com wrote:

Thank you Joerg, in my case, all logs will be indexed by flume through
elasticsearch sink, so, only if flume is power enough to feed 7000eps, es
running on this machine is also power enough to accept?

I am also planning a cluster with 3 node, all of them are configured
as master+data node
And the minimum master node is set to 2. Can I configure my elasticsearch
sinks index to multiple nodes simultaneously? Is this a good idea ti
increase index performance?

Thanks,

Alan

On Thursday, December 25, 2014, joergprante@gmail.com <
joergprante@gmail.com> wrote:

It depends on the power of the indexer program you want to use, you
should benchmark this for yourself.

If you index valuable data, one machine is not enough for fault tolerance.

Jörg

On Thu, Dec 25, 2014 at 3:26 PM, Wang cnwangyong@gmail.com wrote:

Dear all,
I am trying index 7000 messages per second which each message is 1kbyte.
No analyzer and no ttl. I have a machine with 2.0G 24 core cup, 380g
memory, and 300g sas *8. Is this machine powerful enough for this?

Thanks and happy holiday!

Alan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWKJxL4kC_ScZrBqfNskECiq8bqRdSnJunsqxpf%2BjcSOg%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWKJxL4kC_ScZrBqfNskECiq8bqRdSnJunsqxpf%2BjcSOg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEAGY%2BtPM18np3boPy1cZjUH%3DH_tohK-53Ubj50Y%2BzFWA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEAGY%2BtPM18np3boPy1cZjUH%3DH_tohK-53Ubj50Y%2BzFWA%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWq%3DX75DE3uA7uaEbohey8fQtGH%2BW3hsv_eT3U9%3DsW_MA%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAPwW3GWq%3DX75DE3uA7uaEbohey8fQtGH%2BW3hsv_eT3U9%3DsW_MA%40mail.gmail.com?utm_medium=email&utm_source=footer
.

For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoFkLN5CoHX49DxVY_TyfALHJ%2B4zOiBVzvty%3DSYwSc7s1w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.