# Optimizing bulk indexing throughput and memory mapping overhead in large-scale Elasticsearch clusters

**URL:** <https://discuss.elastic.co/t/optimizing-bulk-indexing-throughput-and-memory-mapping-overhead-in-large-scale-elasticsearch-clusters/391343>\
**Category:** Beats\
**Tags:** beats-development\
**Created:** [November 3, 2026, 8:10am UTC](https://discuss.elastic.co/t/optimizing-bulk-indexing-throughput-and-memory-mapping-overhead-in-large-scale-elasticsearch-clusters/391343 "2026-11-03T08:10:01Z")\
**Posts on this page:** 1\
**Page:** 1

<div class="post-metadata">

**Author:** ![softuvo](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/softuvo/32/148534_2.png) [@softuvo](https://discuss.elastic.co/u/softuvo)\
**Post date:** [November 3, 2026, 8:10am UTC](https://discuss.elastic.co/t/optimizing-bulk-indexing-throughput-and-memory-mapping-overhead-in-large-scale-elasticsearch-clusters/391343/1 "2026-11-03T08:10:02Z")

</div>

Hi Elastic Community,

While tuning a high-throughput log ingestion and real-time search pipeline at Softuvo, our team recently analyzed performance bottlenecks involving bulk indexing queues, refresh intervals, and JVM heap usage under heavy concurrent load.

**The Challenge:**

When streaming massive concurrent payloads (handling millions of structured JSON documents per hour), default shard refresh intervals and unoptimized `bulk` request sizing led to occasional thread pool rejections and spikes in garbage collection pauses.

**Steps We Tested:**

1. **Dynamic Refresh Tuning:** Adjusting `index.refresh_interval` from the default `1s` to `30s` during heavy bulk load phases.

2. **Bulk Payload Sizing:** Optimizing `chunk_size` bytes rather than strict document counts to maximize network payload efficiency without hitting circuit breaker limits.

3. **Mapping Optimization:** Explicitly defining mappings to avoid dynamic field mapping overhead and fielddata bloat on high-cardinality text fields.

**Benchmark Results:**

- **Indexing Rate:** ~28% increase in documents indexed per second.

- **Cluster Stability:** Eliminated `es_rejected_execution_exception` errors entirely during peak ingestion windows.

**Minimal Example / Bulk Payload Template:**

JSON

```auto
// Example optimized bulk header configuration
{ "index" : { "_index" : "app-logs-2026.11", "_id" : "custom_id_101" } }
{ "timestamp": "2026-11-03T11:10:00Z", "level": "INFO", "service": "auth-api", "laten

```