We're having a serious issue using Elasticsearch at work without large hosting costs. A recent requirement change bumped us up from $120 to $700 a month. Essentially, this is the issue.
-
We have a catalog of products, and we have companies that can order those products for their employees (We call the companies customers).
-
Customers can either view the product in the customer catalog, or they can add the product to theirs and change some of the options. Because of this, some customers can have a mix of both and so we have only one kind of document: "Product"
Recently, we needed to accommodate the fact that customers can essentially "override" options when they add the product to their own catalog. To do this, we did a naive implementation and just copied over the entire product catalog to each customer (overriding the options as needed). We have thousands of products and thousands of customers, so this ballooned our document size to the tens of millions and isn't tenable on the hosting side. It's also quite slow to build.
There are a few ways I've thought about doing this.
- Only having the public catalog, with an array of objects called
customerOverrides. This way, we aren't making so many documents, but we might have a big nested object field that we have to query on. - Instead of copying the whole public catalog, only copy the ones that have been explicitly added to a customer. This would reduce the document size, but we would have to do two queries to first get the products they've added, and then get the public catalog SANS the ones they added. I haven't found a good "subquery" way to do this in Elasticsearch where the products from a subquery override the products from an outer query.
I can provide more detail if needed, but I am just a little stuck