Hail, guys!
the preliminary basic info:
es version: 6.7
nodes in cluster: 1
Heap size: 30gb
license type: basic
os: Debian 9
The question is about the monitoring-es-6 indices (further: system indices ).
The whole system indices workflow looks fine to me. (system indices creation and rotation after default 7 days)
The thing which is bothering me is the system indices indexing behavior:
Daily docs per day: 17-18 mil docs
Index size daily: 7-8 gb
Ive created a system index pattern for .monitoring-es-6-* and can see the following log source:
{
"_index": ".monitoring-es-6-2019.06.12",
"_type": "doc",
"_id": "xrdOlBLWTcGHErl2mGh7Qw:NqbIcQD6Rsm6vnaBheI2Sg:app-index-2019-06-04-background_load_complete:0:p",
"_version": 360,
"_score": null,
"_source": {
"cluster_uuid": "ZSRwpnBCSvCbXIiwUuM6ZQ",
"timestamp": "2019-06-12T23:59:59.057Z",
"interval_ms": 10000,
"type": "shards",
"source_node": {
"uuid": "Nbt6IcQD6k1L6vnaBheI2Sg",
"host": "10.0.10.10",
"transport_address": "10.0.10.10:9300",
"ip": "10.0.10.10",
"name": "node-master-1",
"timestamp": "2019-06-12T23:59:58.254Z"
},
"state_uuid": "xrdOlBLWTcGHErl2mGh7Qw",
"shard": {
"state": "STARTED",
"primary": true,
"node": "Nbt6IcQD6k1L6vnaBheI2Sg",
"relocating_node": null,
"shard": 0,
"index": "app-index-2019-06-04-background_load_complete"
}
},
"fields": {
"source_node.timestamp": [
"2019-06-12T23:59:58.254Z"
],
"timestamp": [
"2019-06-12T23:59:59.057Z"
]
},
"sort": [
1560383999057
]
}
The thing ive payed attention to - is that the xpack monitoring run through all of my index list and logging it over and over again.
I doubt that its an expected behavior, because ive read in manuals and on the forum that the whole system indices usually doesnt take a lot of resources
For the whole data set (system and app indices) i see
Data size total: 54.6 GB
Docs count total: 136,784,086 (for system indices: 135,079,805| for app indices: 1,671,669 for 2 monthes)
Heap used: 40%
The server is just slightly loaded.
Due to my app and policy i have to keep LOTS (2250 at this moment) of tiny indices for 2 months.
A little bit of app index stats:
Avg index size: 5 mb
Avg docs count per index: 20k
I also have a few (6) bigger indices (300k docs per index, 40mb size per index)