I have a service that apply fine grained access control to ES index document and I am facing some ES limitation / performance issue. For this reason I am not sure if my approach is correct, of if there is a better way to handle that. I am using ES 6.8
I have an Apollo JS server that run a query towards a Postgres DB gathering a list of all the document/resources the user is allowed to access. This is a list of strings and it it pretty long, about 58k strings. This is applied to the query adding a filter to the other filters that are part of the query, if any:
IN: {
["auth_resource_path"]: [...resourceList],
},
an example of a piece of the query body is:
query body: {"size":0,"query":{"bool":{"must":[{"terms":{"auth_resource_path":["/programs/pcdc/projects/20210325/persons/person_156977/subjects/0x92DDDD10F2A2645764CE4E15619924CE","/programs/pcdc/projects/20210223/persons/person_156789/subjects/0xA4CE42BAEAFFD85A5A573F7C0488647D","/programs/pcdc/projects/20210223/persons/person_156791/subjects/0xE23B0F16F4B158D1A417B2B422AEB303","/programs/pcdc/projects/20210223/persons/person_156792/subjects/0xECCB3D6714826A61C09404348C2A9C1F","/programs/pcdc/projects/20210223/persons/person_156793/subjects/0xE24EE86EB9F21939CD169B92BCD78F34","/programs/pcdc/projects/20210223/persons/person_156794/subjects/0x51220822D576DDC4F1F9B68425E8005A","/programs/pcdc/projects/20210223/persons/person_156795/subjects/0x9F76342930F67A310A173B007E3256BD","/programs/pcdc/projects/20210223/persons/person_156796/subjects/0xDC7CFD23C465D23E38FE1245874B8863","/programs/pcdc/projects/20210223/persons/person_156797/subjects/0x6EE4D7DE109189B2C3A322F0D675D15F","/programs/pcdc/projects/20210223/persons/person_156798/subjects/0x3146E12F9120E91DE60FD2F15DCBE315","/programs/pcdc/projects/20210223/persons/person_156799/subjects/0xB6144F3315F8CC556E5C9F953DDABAD7","/programs/pcdc/projects/20210223/persons/person_156800/subjects/0x1CFFE383FB26D75741347B9D0A881EA1","/programs/pcdc/projects/20210223/persons/person_156801/subjects/0x224ACD5DFFFA0403F7CC3DE9278A11F7","/programs/pcdc/projects/20210223/persons/person_156802/subjects/0x4E55992122D9B32CBA7BEFCCCAD2526D","/programs/pcdc/projects/20210223/persons/person_156803/subjects/0x6AE372BFAB833154B799961B8A725B41","/programs/pcdc/projects/20210223/persons/person_156804/subjects/0x13F02A02FE0E1F8A892D43503DF9CD71","/programs/pcdc/projects/20210223/persons/person_156805/subjects/0xB14DDE5D87CB597DA6D97C89BE685074","/programs/pcdc/projects/20210223/persons/person_156806/subjects/0xF3F2708741598DB2CA39D8B501C09717","/programs/pcdc/projects/20210223/persons/person_156807/subjects/0x45BF61D24C666B3C9878EAA5335A7032","/programs/pcdc/projects/20210223/persons/person_156808/subjects/0x2CC7088D4E5A45691B17E750BE080FE7","/programs/pcdc/projects/20210223/persons/person_156809/subjects/0x3A8A95E30C07B394B8623196DB935704","/programs/pcdc/projects/20210223/persons/person_156810/subjects/0xB7A4047EC240A80AF6DDEA908D8B855B","/programs/pcdc/projects/20210223/persons/person_156811/subjects/0x05C74F58C1B84338607C318554707210","/programs/pcdc/projects/20210223/persons/person_156812/subjects/0x56F475FF2C671D510955E5AD69FAB00D","/programs/pcdc/projects/20210325/persons/person_156789/subjects/0x2B1D2E3C4648236211D982AA60BAC9BD","/programs/pcdc/projects/20210325/persons/person_156790/subjects/0xE23B0F16F4B158D1A417B2B422AEB303","/programs/pcdc/projects/20210325/persons/person_156791/subjects/0xECCB3D6714826A61C09404348C2A9C1F","/programs/pcdc/projects/20210325/persons/person_156792/subjects/0xE24EE86EB9F21939CD169B92BCD78F34","/programs/pcdc/projects/20210325/persons/person_156793/subjects/0x51220822D576DDC4F1F9B68425E8005A","/programs/pcdc/projects/20210325/persons/person_156794/subjects/0x9F76342930F67A310A173B007E3256BD","/programs/pcdc/projects/20210325/persons/person_156795/subjects/0xDC7CFD23C465D23E38FE1245874B8863","/programs/pcdc/projects/20210325/persons/person_156796/subjects/0x6EE4D7DE109189B2C3A322F0D675D15F","/programs/pcdc/projects/20210325/persons/person_156797/subjects/0x3146E12F9120E91DE60FD2F15DCBE315","/programs/pcdc/projects/20210325/persons/person_156798/subjects/0xB6144F3315F8CC556E5C9F953DDABAD7","/programs/pcdc/projects/20210325/persons/person_156799/subjects/0x1CFFE383FB26D75741347B9D0A881EA1","/programs/pcdc/projects/20210325/persons/person_156800/subjects/0x224ACD5DFFFA0403F7CC3DE9278A11F7","/programs/pcdc/projects/20210325/persons/person_156801/subjects/0x4E55992122D9B32CBA7BEFCCCAD2526D","/programs/pcdc/projects/20210325/persons/person_156802/subjects/0x6AE372BFAB833154B799961B8A725B41","/programs/pcdc/projects/20210325/persons/person_156803/subjects/0x13F02A02FE0E1F8A892D43503DF9CD71","/programs/pcdc/projects/20210325/persons/person_156804/subjects/0xB14DDE5D87CB597DA6D97C89BE685074","/programs/pcdc/projects/
When the 'resourceList' was about 10k items the query was slow but resolving, but now, with 58k items it runs out of time most of the times.
Do you have any suggestion on the best way to solve this issue? My goal is to give every user their specific list of documents they are allowed to access.