Cluster with a lot of aliases

Elastic Stack Elasticsearch
1

Hey guys,

We at Swiftype use ElasticSearch a lot and absolutely love its power and
flexibility. Thanks for the great product!

We use aliases a lot on our clusters to implement shared indexes for
multiple customers and in some cases we could have hundreds and thousands
of aliases on an index. It was never a huge problem, but recently we've
started seeing more and more strange java memory usage issues (LOTS of
garbage is being generated while processing cluster state changes:
adding/removing indexes and aliases) on a cluster with 800 shards
(700+indexes) and more than 30000 aliases on one one of the indexes.

I do understand that that is most probably a really unusual edge case for
elasticsearch, but still - is there anything you could recommend for
reducing the amounts of garbage generated in memory while processing
cluster state updates in this case?

P.S. I'm trying to create a reproducible test case that would show this
issue on an arbitrary cluster, will let you know how it goes.

--
Oleksiy Kovyrin
http://kovyrin.net/

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

Hey,

thanks for kind words, highly appreciated.

regarding your problems: We are using google guava maps classes (which are
immutable and thus a lot of object creations are going on). There was a
recent change (in the 0.90.7 release), which does not do this anymore in
case of mapping updates, so we have a base of classes we possibly could use
here as well.

May I ask, what your update patterns are? You are adding/removing
indices/aliases all the time and therefore have cluster state updates as
well? How big is your cluster state?

Oh, and we love reproducible test-cases :slight_smile:

--Alex

On Wed, Oct 30, 2013 at 6:33 AM, Oleksiy Kovyrin alexey@kovyrin.net wrote:

Hey guys,

We at Swiftype use Elasticsearch a lot and absolutely love its power and
flexibility. Thanks for the great product!

We use aliases a lot on our clusters to implement shared indexes for
multiple customers and in some cases we could have hundreds and thousands
of aliases on an index. It was never a huge problem, but recently we've
started seeing more and more strange java memory usage issues (LOTS of
garbage is being generated while processing cluster state changes:
adding/removing indexes and aliases) on a cluster with 800 shards
(700+indexes) and more than 30000 aliases on one one of the indexes.

I do understand that that is most probably a really unusual edge case for
elasticsearch, but still - is there anything you could recommend for
reducing the amounts of garbage generated in memory while processing
cluster state updates in this case?

P.S. I'm trying to create a reproducible test case that would show this
issue on an arbitrary cluster, will let you know how it goes.

--
Oleksiy Kovyrin
http://kovyrin.net/

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.