# Parallelizing cluster state listeners **URL:** \ **Category:** Elasticsearch\ **Created:** [May 8, 2019, 11:55am UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165 "2019-05-08T11:55:53Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [May 8, 2019, 11:55am UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/1 "2019-05-08T11:55:53Z")
Based on my understanding the `ClusterApplierService` uses `callClusterStateListeners` to invoke listeners serially which slows down the cluster state publication. Since the action performed by the listeners involve invoking a TransportAction on master to queue it to the pending queue which would eventually be processed serially in batches, does it make sense to parallelize listeners ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 8, 2019, 12:05pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/2 "2019-05-08T12:05:41Z")
> [@Bukhtawar\_Khan](#): > > does it make sense to parallelize listeners Not really, because some listeners depend for their correctness on not running concurrently with other cluster state application activity. Each listener can of course do some of its work asynchronously if needed. ---
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [May 8, 2019, 12:10pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/3 "2019-05-08T12:10:04Z")
But while listeners are running post the cluster state appliers, the cluster state update isn't supposed to happen right. When you say > concurrently with other cluster state application activity do you mean cluster state can be modified concurrently by the listeners ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 8, 2019, 12:33pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/4 "2019-05-08T12:33:26Z")
> [@Bukhtawar\_Khan](#): > > do you mean cluster state can be modified concurrently by the listeners No, the listeners cannot modify the cluster state. I do not really understand what you're asking. Could you be more specific? ---
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [May 8, 2019, 12:41pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/5 "2019-05-08T12:41:14Z")
@DavidTurner I didn't get this part > some listeners depend for their correctness on not running concurrently with other cluster state application activity Can you provide any specifics on the listener and the activity you are referring to here Also do you think we should separate out those listeners that are plain observers and can safely execute concurrently so that atleast some of them can be executed in parallel ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 8, 2019, 1:01pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/6 "2019-05-08T13:01:40Z")
> [@Bukhtawar\_Khan](#): > > Can you provide any specifics on the listener and the activity you are referring to here For instance, `IndicesStore` is responsible for deleting unneeded shard data, but can run into problems if it's doing so concurrently with `IndicesClusterStateService` applying a cluster state that reinstates a shard copy that was relocated away from a node and then back again. ---
**Author:** ![Bukhtawar\_Khan](https://avatars.discourse-cdn.com/v4/letter/b/71c47a/32.png) [@Bukhtawar\_Khan](https://discuss.elastic.co/u/Bukhtawar_Khan)\ **Post date:** [May 8, 2019, 1:47pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/7 "2019-05-08T13:47:25Z")
@DavidTurner Thanks but here is the confusion `IndicesClusterStateService` implements `ClusterStateApplier` while `IndicesStore` implements `ClusterStateListener` and appliers are invoked before the listeners. Is it because the `IndicesStore` calls the applier service My concern is we should allow listeners to execute in parallel if they don't run into conflicts so as to make cluster state publication faster ---
**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\ **Post date:** [May 8, 2019, 2:55pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/8 "2019-05-08T14:55:29Z")
> [@Bukhtawar\_Khan](#): > > My concern is we should allow listeners to execute in parallel if they don't run into conflicts so as to make cluster state publication faster The idea seems unnecessarily complicated. It'd be preferable to investigate the slow listeners and see if they can be sped up or made asynchronous on a case-by-case basis. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 5, 2019, 3:00pm UTC](https://discuss.elastic.co/t/parallelizing-cluster-state-listeners/180165/9 "2019-06-05T15:00:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.