I have a cluster setup where some nodes can experience high system load for a specific index.
Index
The index is time-based, and is named as such:
index-2021.03
index-2021.04
index-2021.05
In other words, I begin a new index with each month.
Index Settings
The index is setup with
- 5 primaries
- 1 replica.
In other words, there are 10 shards total for the index.
Each index is approximately 500GB (that is, each shard is approximately 50GB)
Data Nodes
There are 7 hot data nodes within the cluster. These are the only nodes that contain the indices mentioned above.
Config
Each data node has
- 32GB Ram
- 640GB SSD storage
- Heap: 16GB
- Number of shards per node: ~248
Search
The search performs aggregation against the indices. They are filtered. That is, each query contains a bool filter with a set time range. (the rest of the query is truncated from this post as it is quite long and contains complex aggregations)
{
"query": {
"bool": {
"filter": [
{
"range": {
"timestamp": {
"gte": "now-7d/h",
"lte": "now/h",
"format": "epoch_millis"
}
}
}
]
}
}
}
Search is made against index-*
Expectations
As I understand it, if I perform a search this way and each index only contains documents within that time range, then it shouldn’t matter how large the index is. My expectation is that the shards from the older indices will simply be ignored in these search queries because of the timestamp filter.
Reality
The system load got excessively high for many days. However, once I have deleted some old indices, the system load returns to normal, e.g.:
DELETE index-2021.03
DELETE index-2021.04
Questions
- What could contribute this?
- If I understand correctly, each search is performed against 5 shards at any time — though if there are more indices, am I searching against more shards because there are only 7 data nodes total?
- Should I use more data nodes? (though that seems counter intuitive)
- Should I split the index up into smaller indices? Your docs recommend 50GB per shard, which these shards meet
- Should I use beefier data nodes? Is a data node with the twice the specs of what I am currently using going to be better than me running 2 data nodes?