# How many processes can update the same index in parallel?
**URL:** https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051
**Category:** Elasticsearch
**Created:** [May 14, 2021, 5:22pm UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051 "2021-05-14T17:22:50Z")
**Posts on this page:** 7
**Page:** 1
### Author:  [@Dmitry\_Reshetnik](https://discuss.elastic.co/u/Dmitry_Reshetnik)
#### Post date: [May 14, 2021, 5:22pm UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051/1 "2021-05-14T17:22:50Z")
I have an index in ES that contains 50-100 fields.
I'm getting all those fields from 10-15 different data sources (as partial documents).
For each data source I want to configure an indexer process(es) that will update the index in ES with the new data.
1. Are there any limitations on parallel indexing for ES?
2. Can I have for example 5 instances of each indexer (50-100 instances working in parallel within the same index)?
3. How to calculate the capacity of the index?
4. Is that possible to corrupt the index data by having too many indexers in parallel?
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 15, 2021, 6:14am UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051/2 "2021-05-15T06:14:59Z")
How large is the data set in terms of document count?
How frequently is each individual document likely to be updated?
Are you going to use bulk updates?
Which version of Elasticsearch are you using?
Are you updating individual documents directly by ID?
---
### Author:  [@Dmitry\_Reshetnik](https://discuss.elastic.co/u/Dmitry_Reshetnik)
#### Post date: [May 15, 2021, 10:56am UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051/3 "2021-05-15T10:56:01Z")
> [@Christian\_Dahlqvist](#):
>
> How large is the data set in terms of document count?
250-500M
> [@Christian\_Dahlqvist](#):
>
> How frequently is each individual document likely to be updated?
permanently, but the periodicity is different. some parts will be updated permanently, some of them once per day. over the 24h there is a low chance that more than 10M documents will be updated.
> [@Christian\_Dahlqvist](#):
>
> Are you going to use bulk updates?
low chance because I'm using Kafka as a source of item changes.
> [@Christian\_Dahlqvist](#):
>
> Which version of Elasticsearch are you using?
7.6.1 with Lucene 8.4
> [@Christian\_Dahlqvist](#):
>
> Are you updating individual documents directly by ID?
yes
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 15, 2021, 11:22am UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051/4 "2021-05-15T11:22:03Z")
> [@Dmitry\_Reshetnik](#):
>
> some parts will be updated permanently
Does this mean several updates per second? If not, what periodicity does it translate to?
I do not have a lot of recent experience with high-update use cases, but can provide some pointers based on what I have seen here in the forum.
1. Updating documents without using bulk requests can lead to a lot small segments being generated, which is inefficient and can have huge negative impact on performance. I believe some improvements might have been made in later versions so would recommend that you upgrade to the latest version.
2. Elasticsearch have never been optimised for very frequent updates, so if you have e.g. counters that are updated several times per second you may be better off aggregating these outside Elasticsearch and updating periodically.
3. I do not think you will see corruption if you have highly concurrent updates, but throughput may be poor and you might get errors due to version conflicts. You need to benchmark to tell for sure as it is a somewhat unusual use case.
---
### Author:  [@Dmitry\_Reshetnik](https://discuss.elastic.co/u/Dmitry_Reshetnik)
#### Post date: [May 15, 2021, 11:36am UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051/5 "2021-05-15T11:36:50Z")
Thank you for the response. We had it working in Azure Search with 9-12 concurrent permanent indexers. The fastest stream was processed in 5 single thread processes with the constant updates queue.
Now we are looking to change the scaling approach that may lead to more indexers in parallel.
---
### Author:  [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)
#### Post date: [May 15, 2021, 1:31pm UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051/6 "2021-05-15T13:31:43Z")
What did that setup look like? How many indices and shards? What throughput did you get to? What is now you target?
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 12, 2021, 1:32pm UTC](https://discuss.elastic.co/t/how-many-processes-can-update-the-same-index-in-parallel/273051/7 "2021-06-12T13:32:25Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.