One suggestion can be to have a second index if you expect to have thousands of records and often changes.
You keep your user index as it with create_time and title field but you remove access_group
You will save your group and user relation in a different index.
Something like database but without relation constraint.
Merit: you can list all the group and paginate, you can search and more easily and it will be faster (depend on your request).
Demerit: you may need to make 2 requests, one to check the group and one to get the detail of the user, depends on the context.
You can also duplicate your data and keep the list in the access_group field same as you have now and have the list in parallel for the other search, but you need to be careful and maintain 2 index. It can work depend on your constraint and your code.
I use this way to manage tags in blogs and so far I didn't have problem.