# ECE XFS Quota Question **URL:** \ **Category:** Elastic Cloud Enterprise (ECE)\ **Created:** [January 18, 2019, 1:11am UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718 "2019-01-18T01:11:01Z")\ **Posts on this page:** 19\ **Page:** 1
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 18, 2019, 1:11am UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/1 "2019-01-18T01:11:01Z")
Hi, I'm testing ECE 2.0 to see if this is a better solution for our current es cluster management work. I see following warning logs in the cluster log. `[2018-11-02 16:20:59,091][WARN][no.found.runner.allocation.elasticsearch.ElasticsearchDockerContainer] Quota path not initialized, creating directory for path: [/mnt/data/elastic/172.31.16.63/services/allocator/containers/elasticsearch/1b01fd74758543d7857b6aa53a54389b/instance-0000000001/data] {"ec_container_kind":"elasticsearch","ec_container_group":"1b01fd74758543d7857b6aa53a54389b","ec_container_name":"instance-0000000001"}` It seems like that XFS Quota does not function normally. And I see some strange display in ece cloud ui and Kibana Monitoring page. For example, I create a cluster with 1GB Mem and 32GB Disk. Then in cloud ui, the es node is shown as 1GB Mem and 32GB Disk. But if I connect to Kibana and see nodes in Monitoring Panel, the content is different. The node is shown as 1GB Mem but 200GB disk which is the total size of the vm I run ece allocator. ![image](https://us1.discourse-cdn.com/elastic/original/3X/5/2/525e61f88bb2c6dd1b784eb2f95a9c290e973ee2.jpeg) ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/9/c9c48c33ae22b6c29a46d8d58f51fa8a7ee983df.jpeg) Maybe this is relative to xfs quota. Please can you give me some advice for this? This seems to be a severe issue which will bring confusion to our users. thanks! ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 18, 2019, 2:00am UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/2 "2019-01-18T02:00:44Z")
I think that's what you get if XFS isn't set up properly. What is your fstab? ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 18, 2019, 2:16pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/3 "2019-01-18T14:16:55Z")
Following up on this with a bit more detail: I have some recollection that we only describe configuring XFS on Ubuntu - [https://www.elastic.co/guide/en/cloud-enterprise/2.0/ece-configure-hosts.html#ece-xfs-setup-trusty](https://www.elastic.co/guide/en/cloud-enterprise/2.0/ece-configure-hosts.html#ece-xfs-setup-trusty) - because it's usually installed/configured by default on RHEL/Centos, but there are some set ups that bypass that ... the key thing to check is whether you have a line like `/dev/xvdg1 /mnt/data xfs defaults,nofail,pquota,prjquota 0 2` corresponding to the data directory Alex ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 21, 2019, 12:30am UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/4 "2019-01-21T00:30:34Z")
Sorry for late reply. I will try your configuration. thanks! ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 22, 2019, 7:38am UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/5 "2019-01-22T07:38:51Z")
@Alex_Piggott I change fstab as you suggest. ![image](https://us1.discourse-cdn.com/elastic/original/3X/8/c/8c545370396201a538861d7945e83b23470b3f86.png) But it does not solve my problem. The node still get the disk size of container host like below. ![image](https://us1.discourse-cdn.com/elastic/original/3X/b/b/bbbff2bb0f3779a265d14769c5cc755e9074f017.jpeg) But at cloud ui, the disk size is as below: ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/a/4a891d04dc15270f7c8e0e735f35589f4e8e4663.jpeg) My docker info is as below: ``` [root@ip-172-31-27-255 ~]# docker info Containers: 20 Running: 20 Paused: 0 Stopped: 0 Images: 7 Server Version: 18.03.1-ce Storage Driver: overlay Backing Filesystem: xfs Supports d_type: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: 773c489c9c1b21a6d78b5c538cd395416ec50f88 runc version: 4fc53a81fb7c994640722ac585fa9ca548971871 init version: 949e6fa Security Options: seccomp Profile: default Kernel Version: 3.10.0-957.1.3.el7.x86_64 Operating System: CentOS Linux 7 (Core) OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 31.26GiB Name: ip-172-31-27-255.cn-north-1.compute.internal ID: GO5W:7PKF:DTBB:7WK2:2Z4Z:QQYX:2ZZQ:OI5R:E7FR:QZRS:XXQK:22IF Docker Root Dir: /mnt/data/docker Debug Mode (client): false Debug Mode (server): false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` Please help! ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 22, 2019, 2:53pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/6 "2019-01-22T14:53:22Z")
Can you gist the output of `_nodes/stats/fs`? Eg I get: ```auto "data": [ { "total_in_bytes": 34359738368, "free_in_bytes": 34347282432, "mount": "QuotaAwareFileStore(/app (/dev/mapper/data))", "path": "/app/data/nodes/0", "type": "xfs", "available_in_bytes": 34347282432 } ``` for a container that has 32GB available via XFS Can you also check the contents of `/app/config/fsquota.properties`? How big is the actual disk incidentally? ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 22, 2019, 2:56pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/7 "2019-01-22T14:56:26Z")
(oh other question .. what steps did you go through to get from "XFS not configured" to "XFS configured"? Eg you may need to reallocate the clusters, I'm not sure how "dynamic" that setting is?) ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 22, 2019, 2:59pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/8 "2019-01-22T14:59:54Z")
Sounds good! I will try to recreate the cluster. Wait a moment! thanks! ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 22, 2019, 3:08pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/9 "2019-01-22T15:08:52Z")
> [@Alex\_Piggott](#): > > \_nodes/stats/fs Still not working 😭 \_nodes/stats/fs info is as below > /app/config/fsquota.properties is as below: ``` [root@ip-172-31-27-255 ~]# docker exec -it ae bash root@ae1af9b2cead:/# cat /app/config/fsquota.properties #Usage in xfs quota #Tue Jan 22 15:03:22 GMT 2019 remaining=9223372036854775807 total=9223372036854775807 ``` The actual disk size is as below: ``` [root@ip-172-31-27-255 ~]# df -h Filesystem Size Used Avail Use% Mounted on /dev/xvda1 300G 6.2G 294G 3% / ``` I changed /etc/fstab and then restart all ece machine. ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 22, 2019, 3:12pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/10 "2019-01-22T15:12:25Z")
@Alex_Piggott I find some quota log in allocator service logs. ``` [2019-01-22 15:03:22,688][INFO][no.found.runner.managers.XFSQuotaManager] Adding quota using project id [XFSProjectName(es,61657d98b1454298aac2fefbb2f00e91,instance-0000000005)] with hard limit: [65536] MB {"ec_container_kind":"elasticsearch","ec_container_group":"61657d98b1454298aac2fefbb2f00e91","ec_container_name":"instance-0000000005"} [2019-01-22 15:03:22,708][WARN][no.found.runner.managers.XFSQuotaManager] Command [[sudo, -n, xfs_quota, -x, -c, limit -p bhard=65536m rtbhard=65536m 11292, /mnt/data]] returned status code [1]] with output: [O: [Setting up project 11292 (path /mnt/data/elastic/172.31.27.255/services/allocator/containers/elasticsearch/61657d98b1454298aac2fefbb2f00e91/in stance-0000000005/data)...], O: [Processed 1 (/etc/projects and cmdline) paths for project 11292 with recursion depth infinite (-1).], O: [Setting up project 11292 (path /mnt/dat a/elastic/172.31.27.255/services/allocator/containers/elasticsearch/61657d98b1454298aac2fefbb2f00e91/instance-0000000005/logs)...], O: [Processed 1 (/etc/projects and cmdline) pa ths for project 11292 with recursion depth infinite (-1).], O: [Setting up project 11292 (path /mnt/data/elastic/172.31.27.255/services/allocator/containers/elasticsearch/61657d9 8b1454298aac2fefbb2f00e91/instance-0000000005/heap_dumps/compressed)...], O: [Processed 1 (/etc/projects and cmdline) paths for project 11292 with recursion depth infinite (-1).] , E: [xfs_quota: cannot set limits: Function not implemented]] {} ``` Hope this can help ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 22, 2019, 3:25pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/11 "2019-01-22T15:25:11Z")
May it is caused by selinux. I will try to disable it and reallocate cluster. ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 22, 2019, 3:34pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/12 "2019-01-22T15:34:21Z")
Looks like it's an XFS side issue ... XFS has some handy command line tools for debugging this sort of thing (eg see [https://unix.stackexchange.com/questions/224606/xfs-directory-quota-doesnt-work](https://unix.stackexchange.com/questions/224606/xfs-directory-quota-doesnt-work) which shows use of `xfs_quota -c report`, and also has a candidate solution) Alex ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 22, 2019, 3:42pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/13 "2019-01-22T15:42:09Z")
That `Function not implemented` is what you get when project quotas are not enabled, so I think we must still be missing some setup What does `mount | grep xfs` return? ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 22, 2019, 3:44pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/14 "2019-01-22T15:44:54Z")
You are right. The quota is not enabled. ``` [root@ip-172-31-27-255 ~]# mount|grep xfs /dev/xvda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota) selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime) ``` This machine is on aws and I have restart it after I change fstab like below. `UUID=8c1540fa-e2b4-407d-bcd1-59848a73e463 / xfs defaults,nofail,pquota,prjquota 0 0` Do you know how to enable quota on aws ec2? Thanks! ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 22, 2019, 5:10pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/15 "2019-01-22T17:10:26Z")
Looks like this might be the issue: [https://help.directadmin.com/item.php?id=557](https://help.directadmin.com/item.php?id=557) see under `If you see "noquota" in the xfs mount options for the / partition` ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 23, 2019, 2:30pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/16 "2019-01-23T14:30:49Z")
@Alex_Piggott I try to mount a dedicated volume to ece node,like below: ``` /dev/xvda1 on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota) /dev/xvdf on /mnt type xfs (rw,relatime,seclabel,attr2,inode64,prjquota) ``` But when I restart docker the es container continues to restarting. Terrible problem. I find logs as below. ``` [root@ip-172-31-27-255 ~]# docker ps|grep fac 72480aa4690f regist***/cloud-assets/elasticsearch:5.6.13-0 "/sbin/entry-point" 23 hours ago Restarting (10) Less than a second ago fac-61657d98b1454298aac2fefbb2f00e91-instance-0000000008 78bbf3883341 registr***/cloud-assets/elasticsearch:5.6.13-0 "/sbin/entry-point" 6 days ago Restarting (10) Less than a second ago fac-a83495d0b60947a28df8edc4aa64f641-instance-0000000003 [root@ip-172-31-27-255 ~]# docker logs --tail 10 72480aa4690f usermod: no changes groupmod: failure while writing changes to /etc/group usermod: no changes groupmod: failure while writing changes to /etc/group usermod: no changes groupmod: failure while writing changes to /etc/group usermod: no changes groupmod: failure while writing changes to /etc/group usermod: no changes groupmod: failure while writing changes to /etc/group ``` I cannot decide what causes this problem. I try to disable selinux but still got this error. As you see, this container continues to restart. It's very strange. Do you have any idea what is going on here? ---
**Author:** ![Alex\_Piggott](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/alex_piggott/32/11053_2.png) [@Alex\_Piggott](https://discuss.elastic.co/u/Alex_Piggott)\ **Post date:** [January 23, 2019, 2:45pm UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/17 "2019-01-23T14:45:23Z")
I've seen this in 2 different cases (one good and one bad) The good one is simply that this is a permissions error (see all the permissions set up required in the install docs eg [https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts.html#ece-xfs-setup-trusty](https://www.elastic.co/guide/en/cloud-enterprise/current/ece-configure-hosts.html#ece-xfs-setup-trusty)) .. let's assume it's that ... what permissions do you have set up? The bad one is when this is a nasty OS/docker incompatibility - I've only seen this with certain pre-built Azure images that had some non-standard modules compiled in, so it's not likely this is the issue. ---
**Author:** ![rockybean](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rockybean/32/11443_2.png) [@rockybean](https://discuss.elastic.co/u/rockybean)\ **Post date:** [January 24, 2019, 1:01am UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/18 "2019-01-24T01:01:01Z")
I solved this problem by reinstall ece with the dedicated disk with quota enabled. Thanks for your time! @Alex_Piggott ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 7, 2019, 1:01am UTC](https://discuss.elastic.co/t/ece-xfs-quota-question/164718/19 "2019-02-07T01:01:10Z")
This topic was automatically closed 14 days after the last reply. New replies are no longer allowed.