Hi,
My case is the following: I have data in elastic indexes. At some stage, I'm running some post-processing on this data using Python, and I have a new field that I want to be able to make queries on.
For example, this is my base index data:
{
"name": "John Doe",
"id": 123
}
{
"name": "Taylor Smith",
"id": 456
}
After the post-processing, I have the favorite book field:
{
"name": "John Doe",
"id": 123,
"favorite book": "Moby-Dick"
}
{
"name": "Taylor Smith",
"id": 456,
"favorite book": "The Great Gatsby"
}
I want to be able now to make queries based on the base fields and the favorite book, for example: "Search for all the people that are called 'John' and their favorite book is 'Moby-Dick'".
To achieve that, currently, I'm creating a new index with the new field, but it takes a lot of time and resources.
Is there a more efficient way to achieve my goal?
I thought about creating another index with just the ID and the calculated favorite book, making a query on both indexes (with the relevant fields for each one), and merging the result on the application side. I'm not using this method because it will be very wasteful, as I can get a lot of data from each query that I will need to filter on my app (for example - there are a lot of people named John, a lot of people that their favorite book is Moby-DIck, but not a lot named John that their favorite book is Moby-Dick).
I would be glad for advice on how can I achieve my goal in a more efficient way.