After Upgrading an on-premise installation of ECE from 3.5.1 to 3.8.1 , all deployments have this warning messsage, “Elastic added temporary capacity to stabilize the deployment”. There are no deployments in this environment that have any overrides enabled. Resizing/Scaling the deployments has no effect on this warning message, and it is on every single depoyment that we have. I have searched various ECE API endpoints for where this message might be coming from, but have not found it.
So, short lived problem, and a big thanks to Elastic support for pointing this out.
The source of the problem was disk quota.
Elastic pointed out that instances were reporting more disk space than they were allocated – that is the symptom. So say it was a 1Gb of ram instance which by default allocates 32Gb of disk? It was showing >1Tb. That pointed me to investigate the disk configuration of the underlying host.
The solution? Make sure disk quota is enabled for your disks file system, ie /dev/mapper/ecevg-datalv01 /mnt/data xfs prjquota,pquota
Specifically prjquota and pquota.
Documentation here, (or filter for your particular Linux OS.)
I know i’m being verbose but maybe it will help someone.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
