the company I work for plans to scale our ES cluster from 20 nodes to about
900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even possible
and can I expect the indexing and search time to be the same, increase or
decrease.
If it is possible, are there any pitfalls that one has to avoid?
Judging by others on the mailing list, I have not heard of a cluster of
that size. Mozilla perhaps, but maybe not even them.
Will all 900 nodes be part of the same cluster? The network chatter might
be large. IMHO, if you can afford 900 nodes, then you can afford to use
elasticsearch's own professional services.
the company I work for plans to scale our ES cluster from 20 nodes to
about 900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even possible
and can I expect the indexing and search time to be the same, increase or
decrease.
If it is possible, are there any pitfalls that one has to avoid?
I've heard of 1000-node SolrCloud indices that worked, so I assume ES may
be OK there, too. I'd expect a good amount of work, careful configuring
and tuning for such a massive system. How much data, what sort of data,
and what sort of query complexity and rate are we talking about here?
On Friday, November 22, 2013 10:28:43 AM UTC-5, Kim Laplume wrote:
Hi,
the company I work for plans to scale our ES cluster from 20 nodes to
about 900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even possible
and can I expect the indexing and search time to be the same, increase or
decrease.
If it is possible, are there any pitfalls that one has to avoid?
Not directly related to ES cluster size, but still relevant to your goal;
You'll want to invest time into something like Puppet/Chef and couple that
with a module such as the ES puppet module, you'll find provisioning,
extending and generally managing your cluster a lot simpler.
I've heard of 1000-node SolrCloud indices that worked, so I assume ES may
be OK there, too. I'd expect a good amount of work, careful configuring
and tuning for such a massive system. How much data, what sort of data,
and what sort of query complexity and rate are we talking about here?
On Friday, November 22, 2013 10:28:43 AM UTC-5, Kim Laplume wrote:
Hi,
the company I work for plans to scale our ES cluster from 20 nodes to
about 900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even
possible and can I expect the indexing and search time to be the same,
increase or decrease.
If it is possible, are there any pitfalls that one has to avoid?
Yes, but you need to be careful not to let puppet/chef restart your nodes without proper coordination. Right now we use puppet for deployment and configuration but manually groom indexes off of each node before bouncing it.
Not directly related to ES cluster size, but still relevant to your goal; You'll want to invest time into something like Puppet/Chef and couple that with a module such as the ES puppet module, you'll find provisioning, extending and generally managing your cluster a lot simpler.
I've heard of 1000-node SolrCloud indices that worked, so I assume ES may be OK there, too. I'd expect a good amount of work, careful configuring and tuning for such a massive system. How much data, what sort of data, and what sort of query complexity and rate are we talking about here?
On Friday, November 22, 2013 10:28:43 AM UTC-5, Kim Laplume wrote:
Hi,
the company I work for plans to scale our ES cluster from 20 nodes to about 900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even possible and can I expect the indexing and search time to be the same, increase or decrease.
If it is possible, are there any pitfalls that one has to avoid?
Thank you all for your feedback, I will take it into consideration
On Sun, Nov 24, 2013 at 11:38 PM, Nik Everett nik9000@gmail.com wrote:
Yes, but you need to be careful not to let puppet/chef restart your nodes
without proper coordination. Right now we use puppet for deployment and
configuration but manually groom indexes off of each node before bouncing
it.
Not directly related to ES cluster size, but still relevant to your goal;
You'll want to invest time into something like Puppet/Chef and couple that
with a module such as the ES puppet module, you'll find provisioning,
extending and generally managing your cluster a lot simpler.
I've heard of 1000-node SolrCloud indices that worked, so I assume ES may
be OK there, too. I'd expect a good amount of work, careful configuring
and tuning for such a massive system. How much data, what sort of data,
and what sort of query complexity and rate are we talking about here?
On Friday, November 22, 2013 10:28:43 AM UTC-5, Kim Laplume wrote:
Hi,
the company I work for plans to scale our ES cluster from 20 nodes to
about 900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even
possible and can I expect the indexing and search time to be the same,
increase or decrease.
If it is possible, are there any pitfalls that one has to avoid?
If you go ahead with this, I know there are a lot of people on the list
(including me) that would be interested in keeping up to date with your
progress.
Thank you all for your feedback, I will take it into consideration
On Sun, Nov 24, 2013 at 11:38 PM, Nik Everett nik9000@gmail.com wrote:
Yes, but you need to be careful not to let puppet/chef restart your nodes
without proper coordination. Right now we use puppet for deployment and
configuration but manually groom indexes off of each node before bouncing
it.
Not directly related to ES cluster size, but still relevant to your goal;
You'll want to invest time into something like Puppet/Chef and couple that
with a module such as the ES puppet module, you'll find provisioning,
extending and generally managing your cluster a lot simpler.
I've heard of 1000-node SolrCloud indices that worked, so I assume ES
may be OK there, too. I'd expect a good amount of work, careful
configuring and tuning for such a massive system. How much data, what sort
of data, and what sort of query complexity and rate are we talking about
here?
On Friday, November 22, 2013 10:28:43 AM UTC-5, Kim Laplume wrote:
Hi,
the company I work for plans to scale our ES cluster from 20 nodes to
about 900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even
possible and can I expect the indexing and search time to be the same,
increase or decrease.
If it is possible, are there any pitfalls that one has to avoid?
Create a speed layer for realtime data if needed. (eg : counter in redis)
--
Laurent Laborde
Bigdata Hacker
On Wednesday, November 27, 2013 9:59:21 PM UTC+1, Mark Walkom wrote:
If you go ahead with this, I know there are a lot of people on the list
(including me) that would be interested in keeping up to date with your
progress.
On 28 November 2013 00:42, Kim Laplume <k...@talkwalker.com <javascript:>>wrote:
Thank you all for your feedback, I will take it into consideration
On Sun, Nov 24, 2013 at 11:38 PM, Nik Everett <nik...@gmail.com<javascript:>
wrote:
Yes, but you need to be careful not to let puppet/chef restart your
nodes without proper coordination. Right now we use puppet for deployment
and configuration but manually groom indexes off of each node before
bouncing it.
Not directly related to ES cluster size, but still relevant to your
goal; You'll want to invest time into something like Puppet/Chef and couple
that with a module such as the ES puppet module, you'll find provisioning,
extending and generally managing your cluster a lot simpler.
I've heard of 1000-node SolrCloud indices that worked, so I assume ES
may be OK there, too. I'd expect a good amount of work, careful
configuring and tuning for such a massive system. How much data, what sort
of data, and what sort of query complexity and rate are we talking about
here?
On Friday, November 22, 2013 10:28:43 AM UTC-5, Kim Laplume wrote:
Hi,
the company I work for plans to scale our ES cluster from 20 nodes to
about 900 or perhaps more, because of a higher data volume.
Does anyone has experience with clusters of that size, is it even
possible and can I expect the indexing and search time to be the same,
increase or decrease.
If it is possible, are there any pitfalls that one has to avoid?
Best Regards,
Kim
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
