Index alias is broken every night by auto(bulk api)

I use heartbeat 7.3.0 to send data to a 7.3.0 ES hosted in elastic cloud.
I have one docker heartbeat deployed in EKS 1.13, with default values.

Everything runs fine until every night at 1:00AM UTC, in which auto(bulk api) deletes and recreates the index, losing the write alias, deleting all data, and creating a new alias with a shortname:

BEFORE:
GET _alias
"heartbeat-7.3.0-2019.09.12-000001" : {
"aliases" : {
"heartbeat-7.3.0" : {
"is_write_index" : true
}
}

AFTER:
GET _alias
"heartbeat-7.3.0" : {
"aliases" : {}
}

All my dashboards are then broken and also the watchers, with a message "Fielddata is disabled on text fields by default. Set fielddata=true in [monitor.id] bla bla bla".
If I delete the shortname index (heartbeat-7.3.0) and the alias, and then restart the heartbeat, everything is running fine again until next day.

I've tried everything but I can't avoid the disaster every night.

Imgur
Imgur
Imgur

Sorry to hear about this, this sounds quite painful.

Do you have any idea what's deleting the alias? That's not normal behavior, and beats doesn't do that itself.

My eyes are on this scheduled nightly job called 'auto (bulk api)'. As it can be seen in one of the screenshots, the job deletes indexes and then create them again.
The thing is that the index has initially a patten heartbeat-{version}-{dd/mm/yyyy}-{sequence}, and then my guess is that something goes wrong and the index end with the patternheartbeat-{version} form, so the deleting+recreating process is not running clean.

I have no way to debug or access the details of that process, and unfortunately elastic support told me that this is out of my support tier.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.