Hello,
We use nested documents to store user profiles and associated user events (the nested documents) in ES. We index events using an upsert plugin which apart from indexing the event also updates summary fields of the profile - potentially from the entire event history of the user.
Indexing performance drops significantly (factor 10) as the number of nested documents per user grows.
We attempted to address this issue by switching to a parent-child data model with the user profile being the parent document and events as children. However, we have to execute a has_child/inner_hits query from inside the plugin to assemble all user events associated with the parent document. The incurred overhead (>1s) far outweighs the advantage gained by not having to update the nested document so this is not an option.
My question:
Aside from adding more hardware, can anything be done to speed up indexing performance on nested documents?
Batching nested documents is not an option since we operate in a real-time scenario. We already took into account advice like [1].
Thanks!
/David