@stephenb @RainTown Thanks for the detailed questions. Sharing the complete setup and workload details below.
1. Elasticsearch Version & Azure Region
2. Elasticsearch Cluster Infrastructure
We have a 3-node Elasticsearch cluster in Azure. Each node is a Master + Data + Ingest node.
Elasticsearch Nodes
Each node has:
JVM heap is configured to 4 GB on each node.
Kibana
We have one separate VM dedicated to Kibana:
-
VM: Standard B1ms
-
CPU: 1 vCPU
-
Memory: 2 GiB
-
Disk: Standard SSD LRS
-
Size: 30 GB
-
IOPS: 500
-
Throughput: 100 MB/s
So the complete setup is:
3 Elasticsearch VMs + 1 Kibana VM = 4 VMs
3. Elasticsearch Docker Setup
Elasticsearch and Kibana are running as Docker containers.
The directory structure on each Elasticsearch node is:
/opt/elasticsearch/
├── Dockerfile
├── docker-compose.yml
└── config/
└── certs/
├── elastic-certificates.p12
└── elastic-stack-ca.p12
The config directory currently contains only the certificates required for Elasticsearch security/TLS.
Dockerfile
FROM docker.elastic.co/elasticsearch/elasticsearch:9.0.8
# Add any custom plugin installations here if needed
docker-compose.yml
services:
elasticsearch:
build:
context: .
dockerfile: Dockerfile
container_name: artha-es-node-1
restart: unless-stopped
environment:
- node.name=artha-es-node-1
- cluster.name=artha-es-cluster
- node.roles=master,data,ingest,remote_cluster_client
- discovery.seed_hosts=10.60.0.5,10.60.0.6
- cluster.initial_master_nodes=artha-es-node-1,artha-es-node-2,artha-es-node-3
- ES_JAVA_OPTS=-Xms4g -Xmx4g
- bootstrap.memory_lock=true
- xpack.security.enabled=true
- xpack.security.transport.ssl.enabled=true
- xpack.security.transport.ssl.verification_mode=certificate
- xpack.security.transport.ssl.keystore.path=certs/elastic-certificates.p12
- xpack.security.transport.ssl.truststore.path=certs/elastic-certificates.p12
- network.publish_host=10.60.0.4
- reindex.remote.whitelist=2x.x.x.x:9200
ulimits:
memlock:
soft: -1
hard: -1
nofile:
soft: 65536
hard: 65536
volumes:
- es-data:/usr/share/elasticsearch/data
- es-logs:/usr/share/elasticsearch/logs
- /opt/elasticsearch/config/certs/elastic-certificates.p12:/usr/share/elasticsearch/config/certs/elastic-certificates.p12:ro
ports:
- "9200:9200"
- "9300:9300"
mem_limit: 10g
memswap_limit: 10g
logging:
driver: json-file
options:
max-size: "50m"
max-file: "3"
volumes:
es-data:
es-logs:
The configuration is the same on the other two nodes, with the corresponding node name and IP changed.
4. Current Index Sizes
The current index sizes are approximately:
index docs.count store.size
artha_jobs 7,714,557 99.7 GB
artha_archived_jobs 41,420,000 97.1 GB
artha_job_content_vectors_v2 1,633,255 19.7 GB
artha_job_content_vectors_us 2,737,271 10.6 GB
artha_job_content_vectors_v3 2,239,325 4.3 GB
artha_job_content_vectors_au 963,064 3.7 GB
artha_job_content_vectors_gb 268,186 1.0 GB
artha_job_content_vectors_de 234,887 959.5 MB
artha_job_content_vectors_in 49,389 764.9 MB
artha_job_content_vectors_it 151,350 658.8 MB
artha_job_vector_orphans_v1 1,288,145 331.3 MB
artha_job_content_vectors_ca 69,064 297.4 MB
artha_job_content_vectors_ae 6,050 24.5 MB
artha_job_content_vectors_es 927 4.4 MB
artha_job_content_vectors_uk 10 271.8 KB
The main workload is on the artha_jobs index.
5. Index Configuration
We have not intentionally changed the Lucene merge settings from their defaults.
We can provide the complete output of:
GET /artha_*/_settings?flat_settings=true
{
"artha_job_content_vectors_it": {
"settings": {
"index.creation_date": "1785866058285",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_it",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "NFf0iFZhTIKSbWPb5NZM_g",
"index.version.created": "9009000"
}
},
"artha_job_vector_orphans_v1": {
"settings": {
"index.creation_date": "1785412992758",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_vector_orphans_v1",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "ibg5cdv6TpSz3CFoGo_hWw",
"index.version.created": "9009000"
}
},
"artha_jobs": {
"settings": {
"index.creation_date": "1784627760955",
"index.number_of_replicas": "1",
"index.number_of_shards": "3",
"index.provided_name": "artha_jobs",
"index.refresh_interval": "60s",
"index.routing.allocation.include._tier_preference": "data_content",
"index.translog.durability": "request",
"index.uuid": "5q_wb0-qQ-m8h4ZzHHo-kQ",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_es": {
"settings": {
"index.creation_date": "1785855848534",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_es",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "5rD8pbovTa6RLi4dO2kw8Q",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_au": {
"settings": {
"index.creation_date": "1785855857310",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_au",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "kki_Wl_pRPmxorf5bboIZg",
"index.version.created": "9009000"
}
},
"artha_archived_jobs": {
"settings": {
"index.creation_date": "1784620154949",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_archived_jobs",
"index.refresh_interval": "60s",
"index.routing.allocation.include._tier_preference": "data_content",
"index.translog.durability": "request",
"index.uuid": "XF3RJK01QQmkJKEL2qocoQ",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_uk": {
"settings": {
"index.creation_date": "1785853260664",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_uk",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "J2cOU5bWQB6NZALpy1RC_w",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_v2": {
"settings": {
"index.creation_date": "1785391216036",
"index.number_of_replicas": "1",
"index.number_of_shards": "3",
"index.provided_name": "artha_job_content_vectors_v2",
"index.refresh_interval": "1s",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "fNVyOWMKTg-Bp4sNwNn4Vg",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_v3": {
"settings": {
"index.creation_date": "1785412914243",
"index.number_of_replicas": "0",
"index.number_of_shards": "3",
"index.provided_name": "artha_job_content_vectors_v3",
"index.refresh_interval": "60s",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "OSN32zR0T4KmtZ-J8by8XQ",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_gb": {
"settings": {
"index.creation_date": "1785843483982",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_gb",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "Ib1tbxh0RJ6hJUsnVXWN7g",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_de": {
"settings": {
"index.creation_date": "1785853278933",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_de",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "ISKYfN6JQpii4hpfCtuDTg",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_us": {
"settings": {
"index.creation_date": "1785838823164",
"index.number_of_replicas": "1",
"index.number_of_shards": "2",
"index.provided_name": "artha_job_content_vectors_us",
"index.refresh_interval": "60s",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "rg1snS3OQgKVWBIQ80n49Q",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_ca": {
"settings": {
"index.creation_date": "1785843484434",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_ca",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "owwvANg4TUi2NPnu-5__pA",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_ae": {
"settings": {
"index.creation_date": "1785870613061",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_ae",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "40s5m0XMTaiEMD3wgOriTQ",
"index.version.created": "9009000"
}
},
"artha_job_content_vectors_in": {
"settings": {
"index.creation_date": "1785837667292",
"index.number_of_replicas": "1",
"index.number_of_shards": "1",
"index.provided_name": "artha_job_content_vectors_in",
"index.refresh_interval": "60s",
"index.routing.allocation.include._tier_preference": "data_content",
"index.uuid": "5R7or5CcRqK6F1QFekdg_w",
"index.version.created": "9009000"
}
}
}
as well as the current segment information from:
GET /_cat/segments/artha_*?v
Total 853 Segments
- Our Use Case
The main index is:
artha_jobs
Each Elasticsearch document represents one job.
A job document contains information such as:
-
Job title
-
Company
-
Salary
-
Location
-
Job description
-
Other job-related fields
Our backend uses this index to search and return jobs to our frontend.
Job Lifecycle
Jobs have an expiration period. Once a job expires, it should no longer be available in the active search results.
Our current flow is:
New Job
↓
artha_jobs
↓
Job expires
↓
Move to artha_archived_jobs
↓
Delete from artha_jobs
↓
Keep archived for some time
↓
Eventually delete from artha_archived_jobs
The ingestion/processing application runs multiple times throughout the day and can ingest around 6–7 million documents per day.
The same application also identifies expired jobs, moves them to the archive index, and deletes them from the active index.
So the workload consists of:
High indexing + updates + deletes + continuous search traffic
7. Segment Merge Issue
This is where we are currently facing the problem.
During periods of heavy indexing, updating, and deleting, we see a significant increase in Lucene segment activity.
During these periods:
-
Disk IOPS increases significantly.
-
CPU usage can reach around 100%.
-
Segment merging continues for a considerable amount of time.
-
Search performance degrades significantly.
Our normal search latency is around:
200–300 ms
During heavy segment merging:
3–4 seconds
Once the merge activity settles down, search performance comes back closer to the normal 200–300 ms range.
Our understanding is that the high volume of indexing, updates, and especially deletes is generating substantial segment/merge activity.
We cannot simply remove deletes from the workflow because expired jobs must be removed from artha_jobs and should not appear in production search results.
We have therefore not changed the merge settings yet because we wanted to understand the underlying issue and get recommendations before tuning Lucene/Elasticsearch settings.
We would especially appreciate guidance on:
-
Whether our current VM/storage configuration is appropriate for this workload.
-
Whether our index/shard architecture should be changed.
-
Whether there are recommended approaches for handling this level of indexing + updates + deletes.
-
Whether we should consider index rollover/time-based indices or another architecture.
-
Which metrics you would recommend monitoring to confirm the merge pressure.
-
Whether we should consider merge-policy tuning, or leave the defaults and address the architecture/hardware first.
The current merging output is also attached for reference.
We are primarily looking for advice on the right architectural approach before making merge-policy changes.
