Monitoring-es behavior

Hail, guys!
the preliminary basic info:
es version: 6.7
nodes in cluster: 1
Heap size: 30gb
license type: basic
os: Debian 9

The question is about the monitoring-es-6 indices (further: system indices ).
The whole system indices workflow looks fine to me. (system indices creation and rotation after default 7 days)
The thing which is bothering me is the system indices indexing behavior:
Daily docs per day: 17-18 mil docs
Index size daily: 7-8 gb
Ive created a system index pattern for .monitoring-es-6-* and can see the following log source:

{
  "_index": ".monitoring-es-6-2019.06.12",
  "_type": "doc",
  "_id": "xrdOlBLWTcGHErl2mGh7Qw:NqbIcQD6Rsm6vnaBheI2Sg:app-index-2019-06-04-background_load_complete:0:p",
  "_version": 360,
  "_score": null,
  "_source": {
    "cluster_uuid": "ZSRwpnBCSvCbXIiwUuM6ZQ",
    "timestamp": "2019-06-12T23:59:59.057Z",
    "interval_ms": 10000,
    "type": "shards",
    "source_node": {
      "uuid": "Nbt6IcQD6k1L6vnaBheI2Sg",
      "host": "10.0.10.10",
      "transport_address": "10.0.10.10:9300",
      "ip": "10.0.10.10",
      "name": "node-master-1",
      "timestamp": "2019-06-12T23:59:58.254Z"
    },
    "state_uuid": "xrdOlBLWTcGHErl2mGh7Qw",
    "shard": {
      "state": "STARTED",
      "primary": true,
      "node": "Nbt6IcQD6k1L6vnaBheI2Sg",
      "relocating_node": null,
      "shard": 0,
      "index": "app-index-2019-06-04-background_load_complete"
    }
  },
  "fields": {
    "source_node.timestamp": [
      "2019-06-12T23:59:58.254Z"
    ],
    "timestamp": [
      "2019-06-12T23:59:59.057Z"
    ]
  },
  "sort": [
    1560383999057
  ]
}

The thing ive payed attention to - is that the xpack monitoring run through all of my index list and logging it over and over again.
I doubt that its an expected behavior, because ive read in manuals and on the forum that the whole system indices usually doesnt take a lot of resources

For the whole data set (system and app indices) i see
Data size total: 54.6 GB
Docs count total: 136,784,086 (for system indices: 135,079,805| for app indices: 1,671,669 for 2 monthes)
Heap used: 40%

The server is just slightly loaded.
Due to my app and policy i have to keep LOTS (2250 at this moment) of tiny indices for 2 months.
A little bit of app index stats:
Avg index size: 5 mb
Avg docs count per index: 20k
I also have a few (6) bigger indices (300k docs per index, 40mb size per index)

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.