# Slow ES ingestion using Dataflow template with partialUpdates
**URL:** https://discuss.elastic.co/t/slow-es-ingestion-using-dataflow-template-with-partialupdates/334039
**Category:** Elasticsearch
**Tags:** ingest-pipeline
**Created:** [May 22, 2023, 5:14pm UTC](https://discuss.elastic.co/t/slow-es-ingestion-using-dataflow-template-with-partialupdates/334039 "2023-05-22T17:14:23Z")
**Posts on this page:** 2
**Page:** 1
### Author:  [@julius11](https://discuss.elastic.co/u/julius11)
#### Post date: [May 22, 2023, 5:14pm UTC](https://discuss.elastic.co/t/slow-es-ingestion-using-dataflow-template-with-partialupdates/334039/1 "2023-05-22T17:14:23Z")
We are using [this template](https://github.com/GoogleCloudPlatform/DataflowTemplates/blob/main/v2/googlecloud-to-elasticsearch/README_BigQuery_to_Elasticsearch.md) to ingest data once a day from BigQuery to Elastic Search.
It creates a dataflow job using the following relevant parameters:
```auto
"usePartialUpdate": "true",
"batchSizeBytes": "5242880",
"bulkInsertMethod": "INDEX",
"maxNumWorkers": "30",
"workerMachineType": "n1-standard-1"
```
Total Index size: 70 million rows.
Job updates daily: 7 million rows.
Index refreshes every 30 minutes.
Index snapshotting happens once a day, outside the ingestion time.
By only changing the parameter `usePartialUpdate` from `False` to `True`, we see a drop from writing around `7,000 records/second` to `1,500 records/second`.
How come sending an update to on one field from a record is slower than sending the entire record to overwrite?
```auto
Elastic cluster size: 180 GB storage | 4 GB RAM | Up to 8 vCPU - Single Zone.
Elastic version: v8.4.3
Elastic is managed through GCP marketplace.
```
BigQuery, Dataflow and Elastic are all in the same `europe-west1` GCP region.
I found a few references from a few years ago which I hope they were fixed by now.
> [@Elasticsearch bulk update is extremely slow](https://discuss.elastic.co/t/elasticsearch-bulk-update-is-extremely-slow/78258/8):
>
> I am running logstash 5.2 and Elasticsearch 5.1. I was actually hoping that the bottleneck might be logstash as it is a lot more easier to fix. But then when I run the exact same logstash configuration with output { null{} } The logstash process works at 36000 lines per second. So it doesnt seem to be the bottleneck here, Or is there something I might be overlooking?
> [@Slow Bulk Updates on 6.2.3](https://discuss.elastic.co/t/slow-bulk-updates-on-6-2-3/125513):
>
> Hello all, i'm following up on [this comment](https://github.com/elastic/elasticsearch/issues/23792#issuecomment-375932071) on github and the following suggestion. I have a cluster of 8 blades, 32G RAM and Xeon Processors. I receive, from a kafka topic, a lot of documents that need to be inserted (or updated, if they're already existing) into an index that is rotated weekly. I use a custom document ID for this purpose. The documents are DEDUPED on a 24h time window. This means that i receive, for each document, at most a single update every day. The document rate is qu…
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [June 19, 2023, 5:15pm UTC](https://discuss.elastic.co/t/slow-es-ingestion-using-dataflow-template-with-partialupdates/334039/2 "2023-06-19T17:15:03Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.