3rd party scoring service


(Alex S.V.) #1

Hello,

My idea is to use 3rd party scoring service (REST), and currently I'd like
to use native scripts and play with NativeScriptFactory.
The approach has many drawbacks.

Here is my problem - assume we have two entities - products and product
prices. I should filter by price.
Price is a complex thing, because it depends on many factors, like request
date, remote user information, custom provided parameters. In case of
regular parent - child relation and has_child query it's too complex and
too slow to implement it using scripting (currently mvel).

Also one more condition - i have not many products - around 25K, and around
25M different base price items (which are basic for future price
calculation).
There are next ideas:

  1. Have a service, which returns exact price for all product by custom
    parameters like. The drawback is - there should be 5 same calls from each
    shard (if 5 by default). In this case it doesn't matter, where base prices
    are stored - in elasticsearch index, in database or in in-memory storage.
  2. Write a code, which operates over child price documents on concrete
    shard. In this case it will generate prices only for all properties from
    particular shard. But I don't know, if I can access shard index or make
    calls to the index from concrete shard in NativeScriptFactory class.

Could you point me the right way?

P.S. Initially I was interested in Redis-Elasticsearch
example http://java.dzone.com/articles/connecting-redis-elasticsearch

Thanks,
Alex

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/893b22dc-1415-475b-8675-596119f4f1f8%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Itamar Syn-Hershko) #2

You should bring the price over to Elasticsearch and not the other way
around. Scoring against an external service is an added friction with huge
performance costs.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Thu, Jul 31, 2014 at 1:50 PM, Alex S.V. alexs.vasilenko@gmail.com
wrote:

Hello,

My idea is to use 3rd party scoring service (REST), and currently I'd like
to use native scripts and play with NativeScriptFactory.
The approach has many drawbacks.

Here is my problem - assume we have two entities - products and product
prices. I should filter by price.
Price is a complex thing, because it depends on many factors, like request
date, remote user information, custom provided parameters. In case of
regular parent - child relation and has_child query it's too complex and
too slow to implement it using scripting (currently mvel).

Also one more condition - i have not many products - around 25K, and
around 25M different base price items (which are basic for future price
calculation).
There are next ideas:

  1. Have a service, which returns exact price for all product by custom
    parameters like. The drawback is - there should be 5 same calls from each
    shard (if 5 by default). In this case it doesn't matter, where base prices
    are stored - in elasticsearch index, in database or in in-memory storage.
  2. Write a code, which operates over child price documents on concrete
    shard. In this case it will generate prices only for all properties from
    particular shard. But I don't know, if I can access shard index or make
    calls to the index from concrete shard in NativeScriptFactory class.

Could you point me the right way?

P.S. Initially I was interested in Redis-Elasticsearch example
http://java.dzone.com/articles/connecting-redis-elasticsearch

Thanks,
Alex

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/893b22dc-1415-475b-8675-596119f4f1f8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/893b22dc-1415-475b-8675-596119f4f1f8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAHTr4ZtVL%2B8Qia7pAcYVt0y0EH8d4U3X7%2Bd3N2v%3DxyFfhjQMdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(Alex S.V.) #3

I think it's acceptable if service responds with 20ms and using some thrift
protocol for example. It's much better then current 500ms - 5s calculations
using elasticsearch scripting.
If we have 25K products than it could be around 300Kb data package from
this service. The risk is in possible broken communication or some
increased latency

Alex

On Thursday, July 31, 2014 1:59:36 PM UTC+3, Itamar Syn-Hershko wrote:

You should bring the price over to Elasticsearch and not the other way
around. Scoring against an external service is an added friction with huge
performance costs.

--

Itamar Syn-Hershko
http://code972.com | @synhershko https://twitter.com/synhershko
Freelance Developer & Consultant
Author of RavenDB in Action http://manning.com/synhershko/

On Thu, Jul 31, 2014 at 1:50 PM, Alex S.V. <alexs.v...@gmail.com
<javascript:>> wrote:

Hello,

My idea is to use 3rd party scoring service (REST), and currently I'd
like to use native scripts and play with NativeScriptFactory.
The approach has many drawbacks.

Here is my problem - assume we have two entities - products and product
prices. I should filter by price.
Price is a complex thing, because it depends on many factors, like
request date, remote user information, custom provided parameters. In case
of regular parent - child relation and has_child query it's too complex and
too slow to implement it using scripting (currently mvel).

Also one more condition - i have not many products - around 25K, and
around 25M different base price items (which are basic for future price
calculation).
There are next ideas:

  1. Have a service, which returns exact price for all product by custom
    parameters like. The drawback is - there should be 5 same calls from each
    shard (if 5 by default). In this case it doesn't matter, where base prices
    are stored - in elasticsearch index, in database or in in-memory storage.
  2. Write a code, which operates over child price documents on concrete
    shard. In this case it will generate prices only for all properties from
    particular shard. But I don't know, if I can access shard index or make
    calls to the index from concrete shard in NativeScriptFactory class.

Could you point me the right way?

P.S. Initially I was interested in Redis-Elasticsearch example
http://java.dzone.com/articles/connecting-redis-elasticsearch

Thanks,
Alex

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/893b22dc-1415-475b-8675-596119f4f1f8%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/893b22dc-1415-475b-8675-596119f4f1f8%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/c61f9637-3de8-4906-a2c4-49055dee2cd5%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(system) #4