Optimizing bulk indexing throughput and memory mapping overhead in large-scale Elasticsearch clusters

Elastic Stack Beats
1

Hi Elastic Community,

While tuning a high-throughput log ingestion and real-time search pipeline at Softuvo, our team recently analyzed performance bottlenecks involving bulk indexing queues, refresh intervals, and JVM heap usage under heavy concurrent load.

The Challenge:

When streaming massive concurrent payloads (handling millions of structured JSON documents per hour), default shard refresh intervals and unoptimized bulk request sizing led to occasional thread pool rejections and spikes in garbage collection pauses.

Steps We Tested:

  1. Dynamic Refresh Tuning: Adjusting index.refresh_interval from the default 1s to 30s during heavy bulk load phases.

  2. Bulk Payload Sizing: Optimizing chunk_size bytes rather than strict document counts to maximize network payload efficiency without hitting circuit breaker limits.

  3. Mapping Optimization: Explicitly defining mappings to avoid dynamic field mapping overhead and fielddata bloat on high-cardinality text fields.

Benchmark Results:

  • Indexing Rate: ~28% increase in documents indexed per second.

  • Cluster Stability: Eliminated es_rejected_execution_exception errors entirely during peak ingestion windows.

Minimal Example / Bulk Payload Template:

JSON

// Example optimized bulk header configuration
{ "index" : { "_index" : "app-logs-2026.11", "_id" : "custom_id_101" } }
{ "timestamp": "2026-11-03T11:10:00Z", "level": "INFO", "service": "auth-api", "laten