Custom Scorer and Custom Analyzer

Elastic Stack Elasticsearch
1

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into elastic search
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting a
custom Similarity java object.

We have a specific type of token that we are building, and we have our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

2

Custom analyzer is simple, and there are plenty of samples, here is one:
GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:
https://github.com/elasticsearch/elasticsearch/tree/master/src/main/java/org/elasticsearch/index/similarity.
You need to implement a custom SimilarityProvider (
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/index/similarity/SimilarityProvider.java),
and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker scott@publishthis.com wrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting a
custom Similarity java object.

We have a specific type of token that we are building, and we have our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

3

Fantastic! Will go take a look and see if there is anything else we
may run into.
Thanks,
Scott

On Jan 6, 1:06 pm, Shay Banon kim...@gmail.com wrote:

Custom analyzer is simple, and there are plenty of samples, here is one:GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:https://github.com/elasticsearch/elasticsearch/tree/master/src/main/j....
You need to implement a custom SimilarityProvider (https://github.com/elasticsearch/elasticsearch/blob/master/src/main/j...),
and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker sc...@publishthis.com wrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting a
custom Similarity java object.

We have a specific type of token that we are building, and we have our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

4

Just for everyone's future searching of the groups.
If we did implement our own Scorer (not a Similarity Object) is it
possible to load that into ES as well? If so, how would we do it.

I am talking about this in Lucene: org.apache.lucene.search.Scorer

Thanks,
Scott

On Jan 6, 1:54 pm, Scott Decker sc...@publishthis.com wrote:

Fantastic! Will go take a look and see if there is anything else we
may run into.
Thanks,
Scott

On Jan 6, 1:06 pm, Shay Banon kim...@gmail.com wrote:

Custom analyzer is simple, and there are plenty of samples, here is one:GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:https://github.com/elasticsearch/elasticsearch/tree/master/src/main/j....
You need to implement a custom SimilarityProvider (https://github.com/elasticsearch/elasticsearch/blob/master/src/main/j...),
and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker sc...@publishthis.com wrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting a
custom Similarity java object.

We have a specific type of token that we are building, and we have our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

5

You can build your own custom query parser, register it with the query
registry (it will need to know how to parse itself from a json/xcontent
format). The query parser returns a Lucene Query which can create its
custom scorer. The nice bit is that once you register a custom query, its
part of the query DSL... . Its not documented how to do that, but if you
need it, I can explain...

On Mon, Jan 9, 2012 at 5:15 PM, Scott Decker scott@publishthis.com wrote:

Just for everyone's future searching of the groups.
If we did implement our own Scorer (not a Similarity Object) is it
possible to load that into ES as well? If so, how would we do it.

I am talking about this in Lucene: org.apache.lucene.search.Scorer

Thanks,
Scott

On Jan 6, 1:54 pm, Scott Decker sc...@publishthis.com wrote:

Fantastic! Will go take a look and see if there is anything else we
may run into.
Thanks,
Scott

On Jan 6, 1:06 pm, Shay Banon kim...@gmail.com wrote:

Custom analyzer is simple, and there are plenty of samples, here is
one:GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:
https://github.com/elasticsearch/elasticsearch/tree/master/src/main/j....
You need to implement a custom SimilarityProvider (
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/j...),
and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker sc...@publishthis.com
wrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting
a
custom Similarity java object.

We have a specific type of token that we are building, and we have
our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES
so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

6

A further question on the similarity object.
Is there any way to set a similarity object on a per field basis?

We have special coord/idf/queryNorm/tf for one field, but all other
fields use the default similarity.

Looks like this may not be possible with lucene? Any hacks for it to
work in elasticsearch?

Scott

On Jan 9, 11:38 am, Shay Banon kim...@gmail.com wrote:

You can build your own custom query parser, register it with the query
registry (it will need to know how to parse itself from a json/xcontent
format). The query parser returns a Lucene Query which can create its
custom scorer. The nice bit is that once you register a custom query, its
part of the query DSL... . Its not documented how to do that, but if you
need it, I can explain...

On Mon, Jan 9, 2012 at 5:15 PM, Scott Decker sc...@publishthis.com wrote:

Just for everyone's future searching of the groups.
If we did implement our own Scorer (not a Similarity Object) is it
possible to load that into ES as well? If so, how would we do it.

I am talking about this in Lucene: org.apache.lucene.search.Scorer

Thanks,
Scott

On Jan 6, 1:54 pm, Scott Decker sc...@publishthis.com wrote:

Fantastic! Will go take a look and see if there is anything else we
may run into.
Thanks,
Scott

On Jan 6, 1:06 pm, Shay Banon kim...@gmail.com wrote:

Custom analyzer is simple, and there are plenty of samples, here is
one:GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:
https://github.com/elasticsearch/elasticsearch/tree/master/src/main/j....
You need to implement a custom SimilarityProvider (
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/j...),
and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker sc...@publishthis.com
wrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting
a
custom Similarity java object.

We have a specific type of token that we are building, and we have
our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES
so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

7

No, its not possible in Lucene, thought the upcoming Lucene version (currently trunk) will allow for it.

On Friday, February 3, 2012 at 9:54 PM, Scott Decker wrote:

A further question on the similarity object.
Is there any way to set a similarity object on a per field basis?

We have special coord/idf/queryNorm/tf for one field, but all other
fields use the default similarity.

Looks like this may not be possible with lucene? Any hacks for it to
work in elasticsearch?

Scott

On Jan 9, 11:38 am, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

You can build your own custom query parser, register it with the query
registry (it will need to know how to parse itself from a json/xcontent
format). The query parser returns a Lucene Query which can create its
custom scorer. The nice bit is that once you register a custom query, its
part of the query DSL... . Its not documented how to do that, but if you
need it, I can explain...

On Mon, Jan 9, 2012 at 5:15 PM, Scott Decker <sc...@publishthis.com (http://publishthis.com)> wrote:

Just for everyone's future searching of the groups.
If we did implement our own Scorer (not a Similarity Object) is it
possible to load that into ES as well? If so, how would we do it.

I am talking about this in Lucene: org.apache.lucene.search.Scorer

Thanks,
Scott

On Jan 6, 1:54 pm, Scott Decker <sc...@publishthis.com (http://publishthis.com)> wrote:

Fantastic! Will go take a look and see if there is anything else we
may run into.
Thanks,
Scott

On Jan 6, 1:06 pm, Shay Banon <kim...@gmail.com (http://gmail.com)> wrote:

Custom analyzer is simple, and there are plenty of samples, here is
one:GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:

https://github.com/elasticsearch/elasticsearch/tree/master/src/main/j....

You need to implement a custom SimilarityProvider (

https://github.com/elasticsearch/elasticsearch/blob/master/src/main/j...),

and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker <sc...@publishthis.com (http://publishthis.com)>
wrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting

a

custom Similarity java object.

We have a specific type of token that we are building, and we have
our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES
so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

8

Thanks, this thread help me a lot.

Here is an example of a custom similarity provider is someone needs it:

-- Tanguy
Twitter: @tlrx

Le vendredi 6 janvier 2012 22:06:58 UTC+1, kimchy a écrit :

Custom analyzer is simple, and there are plenty of samples, here is one:
GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:
https://github.com/elasticsearch/elasticsearch/tree/master/src/main/java/org/elasticsearch/index/similarity.
You need to implement a custom SimilarityProvider (
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/java/org/elasticsearch/index/similarity/SimilarityProvider.java),
and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker scott@publishthis.comwrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like inserting a
custom Similarity java object.

We have a specific type of token that we are building, and we have our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

9

Kimchy,

Can you provide more details about how to register a custom query parser
with elasticsearch?

Best,
Ming

On Monday, January 9, 2012 11:38:27 AM UTC-8, kimchy wrote:

You can build your own custom query parser, register it with the query
registry (it will need to know how to parse itself from a json/xcontent
format). The query parser returns a Lucene Query which can create its
custom scorer. The nice bit is that once you register a custom query, its
part of the query DSL... . Its not documented how to do that, but if you
need it, I can explain...

On Mon, Jan 9, 2012 at 5:15 PM, Scott Decker <sc...@publishthis.com<javascript:>

wrote:

Just for everyone's future searching of the groups.
If we did implement our own Scorer (not a Similarity Object) is it
possible to load that into ES as well? If so, how would we do it.

I am talking about this in Lucene: org.apache.lucene.search.Scorer

Thanks,
Scott

On Jan 6, 1:54 pm, Scott Decker sc...@publishthis.com wrote:

Fantastic! Will go take a look and see if there is anything else we
may run into.
Thanks,
Scott

On Jan 6, 1:06 pm, Shay Banon kim...@gmail.com wrote:

Custom analyzer is simple, and there are plenty of samples, here is
one:GitHub - elastic/elasticsearch-analysis-smartcn: Smart Chinese Analysis Plugin for Elasticsearch.

Custom similarity (not scorer, I assume?) is simple as well, just not
documented. Here is the package where the logic resides:
https://github.com/elasticsearch/elasticsearch/tree/master/src/main/j....
You need to implement a custom SimilarityProvider (
https://github.com/elasticsearch/elasticsearch/blob/master/src/main/j...
),
and set it using the setting: index.similarity.index.type and
index.similarity.search.type (this will override the default ones).

On Fri, Jan 6, 2012 at 8:00 PM, Scott Decker sc...@publishthis.com
wrote:

Hey All,
I had searched in the groups here for this info, but it was always
"ping me to get info" =)

How do we go about inserting a custom scorer into Elasticsearch
(insert our own Similarity java object)
How do we go about inserting a custom analyzer - much like
inserting a
custom Similarity java object.

We have a specific type of token that we are building, and we have
our
own analyzer for breaking the token up correctly, and then using the
custom scorer to then score the matches.

I get how to do this in Lucene, just don't see how we do that in ES
so
that the Lucene indexes use these in ES.

Any help would be much appreciated

Thanks,
Scott

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.