Hello!
I have an issue that we are running out of shards in our elastic cloud cluster.
(lots of errors saying: "this action would add [2] shards, but this cluster currently has [2000]/[2000] maximum normal shards open")
When i look at our data streams i can see that alot of them have 20+ open indecies, which is weird because they each only contains like ~20 mb of data.
I think the default behaviour of the integration that created these datastreams is the cause, so i modified the index template for them and added a component template containing:
{
"index": {
"lifecycle": {
"name": "default_unmanaged_logs"
}
}
}
where default_unmanaged_logs is an ILM policy that sets the rollover in hot storage to default (30 days/50 gb) which then should move it to frozen storage.
When i click on the component template i can see that it is used by all the index templates of the misbehaving data streams, but when i
GET _index_template/logs-misbehaving_index_template
I only get this:
{
"index_templates": [
{
"name": "logs-misbehaving_index_template",
"index_template": {
"index_patterns": [
"misbehaving_index_template-*"
],
"template": {
"settings": {},
"mappings": {
"_meta": {
"package": {
"name": "system"
},
"managed_by": "fleet",
"managed": true
}
}
},
"composed_of": [
"logs@settings",
"logs-system.syslog@package",
"ecs@mappings",
".fleet_globals-1",
".fleet_agent_id_verification-1",
"**company-lifecycle-policy@custom**"
],
"priority": 200,
"_meta": {
"package": {
"name": "system"
},
"managed_by": "fleet",
"managed": true
},
"data_stream": {
"hidden": false,
"allow_custom_routing": false
}
}
}
]
}
where company-lifecycle-policy@custom is the component template containing the ILM configuration i mentioned above.
Should the config from the component template show up inline in the index template above or does it not calculate the actual template?
Do i need to "reindex" the open indecies so that the new config applies?
Maybe i'm looking at it the wrong way and the 20+ open indexes is some other issue.