Hi Team,
Currently we have 2 indices for storing the raw datasets without doing any transformations. These datasets have very nested structure.
This was fine for the current use cases where the only search pattern we had was search by document ID.
With AI use cases and for an AI Analyst to answer the questions appropriately based on the data that's stored in these 2 indices with more complex search and filtering use cases.
What would be your recommendations around the 2 options:
- create a new index and a new microservice that would transform the data or denormalize the data catering to the AI use cases?
- Let the AI python logic do the transformations on-the-fly while answering the questions?
FYI: most of the new query patterns are also not too complex and are simple boolean queries with exact matches, some match phrases, IN queries for multiple doc ids, lt/gt date ranges.
Also had gone through the documentation for ES Transform API but may not be exactly relevant for this use case of mine?
Thanks,
Moni