My idea is to use 3rd party scoring service (REST), and currently I'd like
to use native scripts and play with NativeScriptFactory.
The approach has many drawbacks.
Here is my problem - assume we have two entities - products and product
prices. I should filter by price.
Price is a complex thing, because it depends on many factors, like request
date, remote user information, custom provided parameters. In case of
regular parent - child relation and has_child query it's too complex and
too slow to implement it using scripting (currently mvel).
Also one more condition - i have not many products - around 25K, and around
25M different base price items (which are basic for future price
calculation).
There are next ideas:
Have a service, which returns exact price for all product by custom
parameters like. The drawback is - there should be 5 same calls from each
shard (if 5 by default). In this case it doesn't matter, where base prices
are stored - in elasticsearch index, in database or in in-memory storage.
Write a code, which operates over child price documents on concrete
shard. In this case it will generate prices only for all properties from
particular shard. But I don't know, if I can access shard index or make
calls to the index from concrete shard in NativeScriptFactory class.
You should bring the price over to Elasticsearch and not the other way
around. Scoring against an external service is an added friction with huge
performance costs.
My idea is to use 3rd party scoring service (REST), and currently I'd like
to use native scripts and play with NativeScriptFactory.
The approach has many drawbacks.
Here is my problem - assume we have two entities - products and product
prices. I should filter by price.
Price is a complex thing, because it depends on many factors, like request
date, remote user information, custom provided parameters. In case of
regular parent - child relation and has_child query it's too complex and
too slow to implement it using scripting (currently mvel).
Also one more condition - i have not many products - around 25K, and
around 25M different base price items (which are basic for future price
calculation).
There are next ideas:
Have a service, which returns exact price for all product by custom
parameters like. The drawback is - there should be 5 same calls from each
shard (if 5 by default). In this case it doesn't matter, where base prices
are stored - in elasticsearch index, in database or in in-memory storage.
Write a code, which operates over child price documents on concrete
shard. In this case it will generate prices only for all properties from
particular shard. But I don't know, if I can access shard index or make
calls to the index from concrete shard in NativeScriptFactory class.
I think it's acceptable if service responds with 20ms and using some thrift
protocol for example. It's much better then current 500ms - 5s calculations
using elasticsearch scripting.
If we have 25K products than it could be around 300Kb data package from
this service. The risk is in possible broken communication or some
increased latency
Alex
On Thursday, July 31, 2014 1:59:36 PM UTC+3, Itamar Syn-Hershko wrote:
You should bring the price over to Elasticsearch and not the other way
around. Scoring against an external service is an added friction with huge
performance costs.
On Thu, Jul 31, 2014 at 1:50 PM, Alex S.V. <alexs.v...@gmail.com
<javascript:>> wrote:
Hello,
My idea is to use 3rd party scoring service (REST), and currently I'd
like to use native scripts and play with NativeScriptFactory.
The approach has many drawbacks.
Here is my problem - assume we have two entities - products and product
prices. I should filter by price.
Price is a complex thing, because it depends on many factors, like
request date, remote user information, custom provided parameters. In case
of regular parent - child relation and has_child query it's too complex and
too slow to implement it using scripting (currently mvel).
Also one more condition - i have not many products - around 25K, and
around 25M different base price items (which are basic for future price
calculation).
There are next ideas:
Have a service, which returns exact price for all product by custom
parameters like. The drawback is - there should be 5 same calls from each
shard (if 5 by default). In this case it doesn't matter, where base prices
are stored - in elasticsearch index, in database or in in-memory storage.
Write a code, which operates over child price documents on concrete
shard. In this case it will generate prices only for all properties from
particular shard. But I don't know, if I can access shard index or make
calls to the index from concrete shard in NativeScriptFactory class.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.