# Multiple cluster state copies in memory VS many aliases

**URL:** <https://discuss.elastic.co/t/multiple-cluster-state-copies-in-memory-vs-many-aliases/14960>\
**Category:** Elasticsearch\
**Created:** [December 19, 2013, 9:24pm UTC](https://discuss.elastic.co/t/multiple-cluster-state-copies-in-memory-vs-many-aliases/14960 "2013-12-19T21:24:37Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![Oleksiy\_Kovyrin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/oleksiy_kovyrin/32/1901_2.png) [@Oleksiy\_Kovyrin](https://discuss.elastic.co/u/Oleksiy_Kovyrin)\
**Post date:** [December 19, 2013, 9:24pm UTC](https://discuss.elastic.co/t/multiple-cluster-state-copies-in-memory-vs-many-aliases/14960/1 "2013-12-19T21:24:37Z")

</div>

Hey guys,

I'm not sure if you could do anything about the issue, but still, I wanted  
to make sure you are aware of it. So, here is the story:

I work for [Swiftype.com](http://Swiftype.com). Due to the nature of our business and the way we  
implement user search engines as separate or aggregated indexes in ES and  
use per-customer aliases to route requests to an appropriate place. So, we  
end up with clusters that have ~1000 shards and could easily have 50-150k  
aliases. ES holds up really well under load and the only issue we have  
which has been the major pain point for us for the last 4 months is the way  
ES handles cluster state changes.

Every time we add/remove an alias or an index, ES generates gigabytes of  
garbage in java memory. Adding more java heap does not really help, because  
then those piles of garbage start causing very long old gen pauses and that  
is really painful for us since our clusters are constantly under load and  
having them stop for seconds to collect garbage is unacceptable.

For months we've been building all kinds of bandages to mitigate the  
effects of this issue (like creating an external per-cluster locking system  
to avoid consurrent cluster state changes and other crazy stuff like that)  
and trying to figure out why was it was happening without much luck.

Yesterday we've added a master-only (no data) node to one of our small  
clusters and gave it 4G of heap (with new ratio = 3). Even during the  
process of joining the cluster it went to GC for 5-10 sec multiple times,  
every time failing to join becuase of timeouts. After many retries it'd  
finally join and old gen would keep steady at ~1.2Gb (out of 3Gb we gave  
it) for hours and hours. But then, when some users would start  
creating/changing their indexes too frequently (once every few seconds),  
the server would just go crazy and end up in a Full GC trashing mode.

I've made a java heap dump from this server today and since there is no  
Lucene crap in it (as it usually is on larger instances with data), it was  
REALLY obvious now where the problem is: in my dump I see our cluster  
MetaData object size is ~681Mb and we have 6 (!) copies of it live in  
memory:

- one copy in current cluster metadata
- one copy in a running InternalClusterService$UpdateTask
- and 4 copies in InternalClusterService$UpdateTask instances in the  
blocking queue for update service.

I understand that we could add many more gigabytes of RAM and that would  
help us to handle such "surges" of cluster state updates, but I'm afraid  
that would just cost us a lot in GC pauses and would not really solve the  
issues we're having on data nodes anyways.

So, I was wondering if you have any suggestions on how can we mitigate this  
issue in the short term (aside from not creating so many aliases, but that  
does it not really possible for us at the moment) and if you have any ideas  
of how it could be solved in the server code in the long run.

I'd really appreciate any help!

P.S. We're on 0.90.5. If there are any fixes around cluster state handling  
in the newer versions, please let me know. We're going to be upgrading our  
clusters very soon and if there is a chance new release would help with GC  
issues, I'd bump up the priority of the upgrade task in our task list.

--  
Oleksiy Kovyrin  
Head of Technical Operations

> **[Swiftype: Enterprise Search & Site Search Platform](https://Swiftype.com/)**
>
> Swiftype is a cloud-based search platform that provides all the tools you need to create fantastic search experiences.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/8972fd4b-44eb-4188-9831-6469bc8e216d%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/8972fd4b-44eb-4188-9831-6469bc8e216d%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 2:00am UTC](https://discuss.elastic.co/t/multiple-cluster-state-copies-in-memory-vs-many-aliases/14960/2 "2017-07-06T02:00:07Z")

</div>