Index-wise conditions in Kibana watchers

[ErGeek (Ereek)]

Hi All,

We are using Elasticsearch watchers for getting the total data ingested from each index. But since we are having different ILMs for each index, we are not able to get the average data ingested for a particular day.

For example, the index-1 is having 7 days of ILM period, while index-2 is having 10 days of ILM. Could anyone please suggest on how to have the separate conditions for each index?

Please find the existing watcher below:

{
"trigger" : {
"schedule" : {
"interval" : "2m"
}
},
"input": {
"http": {
"request": {
"scheme": "https",
"host": "####",
"port": 9243,
"method": "get",
"path": "_cat/indices",
"params": {
"format": "yaml",
"human": "true",
"bytes": "mb"
},
"headers": {},
"auth": {
"basic": {
"username": "elastic",
"password": "***"
}
}
}
}
},
"condition": {
"script": {
"source": "return true;",
"lang": "painless"
}
},
"actions": {
"email_administrator": {
"transform": {
"script": {
"source": """
ctx.payload.total = (ctx.payload.data.stream().collect(Collectors.summarizingDouble(e -> Double.parseDouble(e['store.size']))).sum)/1024;
return ctx.payload;
""",
"lang": "painless"
}
},
"email": {
"profile": "standard",
"to": [
"myemail@abc.com"
],
"subject": "Index-wise data",
"body": {
"html": """

Index-wise data

{{#ctx.payload.data}} {{/ctx.payload.data}}

IndexName	StoreSize
{{index}}	{{store.size}} MB
TOTAL STORE SIZE	{{ctx.payload.total}} GB

"""
}
}
}
}
}

[Luca_Belluccini]

This message has been answered via DM.
Snippet on how to achieve this:

ctx.payload.data = ctx.payload.data.stream().map(e -> {
if (e['index'].startsWith('something')) { # it is also possible to use `contains(...)`
  e.average_size_per_day = Double.parseDouble(e['store.size']) / 10.0f;
} else if (e['index'].startsWith('somethingelse')) {
  e.average_size_per_day = Double.parseDouble(e['store.size']) / 7.0f;
} else {
  // do something else
}
return e; # this was missing
}).collect(Collectors.toList());

Obviously this will not work if the index rolled over due to its size and not due to age.
The only way to do this would be to look in the index settings of each index and check the creation date.

We have a public issue opened for an enhancement to offer this out of the box at Show per day size stats for ILM indices · Issue #100009 · elastic/kibana · GitHub