# X-Pack role cannot be created or modified **URL:** \ **Category:** Elasticsearch\ **Created:** [July 18, 2017, 11:11pm UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670 "2017-07-18T23:11:50Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![mc2000](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@mc2000](https://discuss.elastic.co/u/mc2000)\ **Post date:** [July 18, 2017, 11:11pm UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/1 "2017-07-18T23:11:50Z")
Hi, I'm experiencing issue when creating a new role which results in the following error message: > {"error":{"root\_cause":[{"type":"illegal\_state\_exception","reason":"role cannot be created or modified as service cannot write until template and mappings are up to date"}],"type":"illegal\_state\_exception","reason":"role cannot be created or modified as service cannot write until template and mappings are up to date"},"status":500} Message is quire cryptic and hard to say what is actually going on. Would appreciate any suggestions. Experimenting with x-pack on existing cluster (5.1.1) which have a bunch of existing indexes already. Request: > POST /\_xpack/security/role/my\_new\_role -d '{ > "cluster": ["monitor"], > "indices": [ > { > "names": ["\*"], > "privileges": ["monitor", "delete\_index"] > } > ] > }' Thanks. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 19, 2017, 12:34am UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/2 "2017-07-19T00:34:12Z")
> [@](#): > > service cannot write until template and mappings are up to date This usually indicates that there is an Elasticsearch version upgrade that hasn't completed yet. When an upgrade is in progress, particularly if it's a rolling upgrade, there are restrictions on updates to security users/roles so that nodes with different versions don't create incompatible changes. Occasionally that upgrade logic can cause issues. To diagnose what's going on, can you walk through the following steps: - Does your cluster have multiple nodes? - If so, please double check that they are all running the exact same version: Run `GET /_nodes/_all/-` and check the `"version"` field for each node is the same. - Check the versions of your security mappings in both the index and template: - Template: Run `GET /_template/security-index-template` and check the `_meta.security-version` for each type in the `mappings` section. - Index: Run `GET /.security/_mappings` (you will need to run this as a superuser, such as the builtin `elastic` user) and check the `_meta.security-version` for each type. They should all match the version that your cluster is running. We should be able to work out what's going on from that. ---
**Author:** ![mc2000](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@mc2000](https://discuss.elastic.co/u/mc2000)\ **Post date:** [July 19, 2017, 10:40am UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/3 "2017-07-19T10:40:34Z")
I followed steps suggested in your response. Note that I've disabled x-pack temporarily but I presume that it should have no impact on tests below? > Does your cluster have multiple nodes? Yes. There is several nodes in the cluster and all of them run same exact version. `"version": "5.1.1"` in this case. > Template: Run GET /\_template/security-index-template and check the \_meta.security-version for each type in the mappings section. Again, all of the types in the mappings sections have the same security-version: `"_meta": { "security-version": "5.1.1" }` > Index: Run GET /.security/\_mappings and check the \_meta.security-version for each type. Similarly here, all types in the index mappings seem to have the same version matching version of the cluster: `"_meta": { "security-version": "5.1.1" }` Thanks. ---
**Author:** ![mc2000](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@mc2000](https://discuss.elastic.co/u/mc2000)\ **Post date:** [July 20, 2017, 2:14pm UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/4 "2017-07-20T14:14:25Z")
Is there anything else I could do to try and figure out why role cannot be created? Could removing `security-index-template` and `.security` index itself help before re-enabling x-pack? Any suggestions will be greatly appreciated. Thanks. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 21, 2017, 7:19am UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/5 "2017-07-21T07:19:00Z")
Sorry for not getting back to you sooner. Removing the `.security` index might help, but without knowing exactly what's causing your problem, it's hard to say for sure. Can you try turning on `DEBUG` for `org.elasticsearch.xpack.security.authz.store.NativeRolesStore` and check for log messages during startup? - Turn on debugging by adding this line to your `elasticsearch.yml` ```auto logger.org.elasticsearch.xpack.security.authz.store.NativeRolesStore: DEBUG ``` - Then restart the node that you are testing against (you don't need to restart the whole cluster, just the one that you're sending requests to) - Check `elasticsearch.log` for messages with `[NativeRolesStore]` in them. You should see something like: > security template [security-index-template] does not exist or is not up to date, so service cannot start _or_ > mapping for security index not up to date, so service cannot start There should be other messages too - if you can post all the relevant ones here, or send them to me in a private message, we should be able to get to the bottom of this. ---
**Author:** ![mc2000](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@mc2000](https://discuss.elastic.co/u/mc2000)\ **Post date:** [July 27, 2017, 11:05pm UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/6 "2017-07-27T23:05:51Z")
I've re-enabled x-pack, added `logger.org.elasticsearch.xpack.security.authz.store.NativeRolesStore: DEBUG` in config and redeployed the cluster. First thing I noticed is that I could no longer authenticate - it seem that once `.security` index gets created (first time I rolled x-pack out) and x-pack is subsequently turned off and then re-enabled it somehow confuses the system. I had to disable x-pack again to skip auth and drop `.security` index and followed up with redeployment of x-pack enabled version - this allowed me to authenticate with default `elastic` user again. I subsequently performed all the checks as per your instructions above, and can confirm that 1. All nodes in the cluster run the same version `5.1.1`. 2. `_meta.security-version` for both `security-index-template` template and `.security` index mappings have same version matching cluster nodes: `5.1.1`. Next, attempt to create a new role appeared to be successful this time: > curl -XPOST -k -u elastic:changeme [https://elasticsearch:9200/\_xpack/security/role/test\_role](https://elasticsearch:9200/_xpack/security/role/test_role) -d ' > { "cluster": ["monitor"], > "indices": [ > { > "names": ["\*"], > "privileges": ["monitor", "delete\_index"] > } > ] > }' result: > {"role":{"created":true}} And corresponding Elasticsearch logs: > [2017-07-27T21:38:44,405][INFO][o.e.c.m.MetaDataCreateIndexService] [es-master-0] [.security] creating index, cause [auto(index api)], templates [security-index-template, template\_2, template\_1], shards [1]/[0], mappings [kubernetes, role, reserved-user, user] > [2017-07-27T21:38:44,765][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:44,828][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:44,841][INFO][o.e.c.m.MetaDataUpdateSettingsService] [es-master-0] updating number\_of\_replicas to [9] for indices [.security] > [2017-07-27T21:38:44,983][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:45,053][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:45,113][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:45,181][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:45,272][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:45,274][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] invalidating role [test\_role] in cache > [2017-07-27T21:38:45,464][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:38:45,511][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > ... Listing of all roles returned expected results: > curl -k -u elastic:changeme [https://elasticsearch:9200/\_xpack/security/role](https://elasticsearch:9200/_xpack/security/role) > { > [... default roles ...] > "test\_role": { > "cluster": [ > "monitor" > ], > "indices": [ > { > "names": [ > "\*" > ], > "privileges": [ > "monitor", > "delete\_index" > ] > } > ], > "run\_as": , > "metadata": {} > } > } All good so far, however when creating a user: > curl -XPOST -k -u elastic:changeme [https://elasticsearch:9200/\_xpack/security/user/test\_user](https://elasticsearch:9200/_xpack/security/user/test_user) -d ' > { > "password" : "changeme", > "roles" : ["test\_role"], > "full\_name" : "Test User", > "email" : "[testuser@example.com](mailto:testuser@example.com)" > }' result: > {"error":{"root\_cause":[{"type":"illegal\_state\_exception","reason":"user cannot be created or changed as the user service cannot write until template and mappings are up to date"}],"type":"illegal\_state\_exception","reason":"user cannot be created or changed as the user service cannot write until template and mappings are up to date"},"status":500} Elasticsearch log: > [2017-07-27T21:41:53,472][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:41:53,564][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > [2017-07-27T21:41:53,618][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-master-0] mapping for security index not up to date, so service cannot start > ... The content of `.security` index: > curl -k -u elastic:changeme [https://elasticsearch:9200/.security/\_search](https://elasticsearch:9200/.security/_search) > {"took":30,"timed\_out":false,"\_shards":{"total":1,"successful":1,"failed":0},"hits":{"total":1,"max\_score":1.0,"hits":[{"\_index":".security","\_type":"role","\_id":"test\_role","\_score":1.0,"\_source":{"cluster":["monitor"],"indices":[{"names":["\*"],"privileges":["monitor","delete\_index"]}],"run\_as":,"metadata":{}}}]}} Worth noting that existing cluster on which I'm trying to enable x-pack was upgraded from version 2.4.1 to 5.1.1. Could this be a reason? I also tried a brand new cluster deployment (no pre-existing indexes) and was able to successfully create role and user, with the relevant logs below: > [2017-07-27T16:26:59,524][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-client-0] native roles store waiting until gateway has recovered from disk > [2017-07-27T16:26:59,524][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-client-0] security template [security-index-template] does not exist or is not up to date, so service cannot start > [2017-07-27T16:27:05,777][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-client-0] security template [security-index-template] does not exist or is not up to date, so service cannot start > [2017-07-27T16:27:05,777][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-client-0] security template [security-index-template] does not exist or is not up to date, so service cannot start > [2017-07-27T16:27:05,777][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-client-0] security template [security-index-template] does not exist or is not up to date, so service cannot start > [2017-07-27T16:27:06,261][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-client-0] security index [.security] does not exist, so service can start > [2017-07-27T16:28:12,320][DEBUG][o.e.x.s.a.s.NativeRolesStore] [es-client-0] invalidating role [my\_role] in cache > [2017-07-27T16:28:46,441][INFO][o.e.x.s.a.u.TransportPutUserAction] [es-client-0] added user [my\_user] Any ideas on how to debug it further will be greatly appreciated. Thanks. ---
**Author:** ![TimV](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/timv/32/13162_2.png) [@TimV](https://discuss.elastic.co/u/TimV)\ **Post date:** [July 28, 2017, 3:14am UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/7 "2017-07-28T03:14:06Z")
> [@mc2000](#): > > Any ideas on how to debug it further will be greatly appreciated. Thanks. Is there anything left to resolve? It appears that somewhere between: - Restarting the nodes - Disabling + enabling X-Pack - Dropping and recreating `.security` everything has gone into a working state. My _guess_ is that the component that tracks the version of your security index ended up in an incorrect state and the restart fixed it. That component has been completely changed in recent versions to try and overcome those sorts of problems. If there's something left to debug, I'm happy to help but it does look like you have a working system now. ---
**Author:** ![mc2000](https://avatars.discourse-cdn.com/v4/letter/m/f475e1/32.png) [@mc2000](https://discuss.elastic.co/u/mc2000)\ **Post date:** [July 28, 2017, 7:51am UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/8 "2017-07-28T07:51:27Z")
I'd like to enable that on existing cluster with bunch of pre-existing indexes but this proves to be difficult. Starting from scratch (same cluster version) is working but I'd rather have it enabled on main cluster. You mentioned that x-pack changed in recent versions to overcome some of the issues above. Would you advise to try and upgrade version of the stack to the latest? I presume there was no major compatibility issues between 5.1.1 and current 5.5.1. Thanks ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 25, 2017, 7:51am UTC](https://discuss.elastic.co/t/x-pack-role-cannot-be-created-or-modified/93670/9 "2017-08-25T07:51:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.