AllTermQueryParser?

Hi,
I am looking for the way how to use token payloads in ES, pretty much same
way as:
in DelimitedPayloadTokenFilter, described
at http://searchhub.org/2009/08/05/getting-started-with-payloads/

I see in the ES codebase almost everything I think I need.
package is: package org.elasticsearch.common.lucene.all

There is AllTokenStream for indexing and AllTermQuery for querying, but I
guess AllTermsQueryParser is missing somehow, or I am missing something?

In other words in my query I need to instruct ES to use AllTermQuery and
for that I understand I need AllTermQueryParser, but there is no such thing.

My questions are:

  1. Is there any reason why such query parser is not implemented?
  2. What would be the best way to implement such query parser?
  3. Is there any better way to implement this use case?

Thanks,
Milan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi again,
still haven't found the answer on how to tell ES to use:

AllTokenStream in the analysis phase and AllTermQuery in the query dsl so
that:

DOC1 has :content" field: "The quick|2.0 red|2.0 fox|10.0 jumped|5.0 over
the lazy|2.0 brown|2.0 dogs|10.0"
DOC2 has "content" field: "The quick|3.0 red|2.0 fox|10.0 jumped|5.0 over
the lazy|2.0 brown|2.0 dogs|10.0"

float values (score boosts) are added in the token payload. I understand
that AllTokenStream does that, but how do I tell that in index call?

Then what do I need to do to get DOC2 scored better than DOC1 when I search
for "red" in "content field"?
I understand I need to use AllTermQuery, but how do I do that?

Thanks in advance,
Milan

On Wed, Jun 12, 2013 at 2:15 PM, Milan Agatonovic <
milan.agatonovic@gmail.com> wrote:

Hi,
I am looking for the way how to use token payloads in ES, pretty much same
way as:
in DelimitedPayloadTokenFilter, described
at http://searchhub.org/2009/08/05/getting-started-with-payloads/

I see in the ES codebase almost everything I think I need.
package is: package org.elasticsearch.common.lucene.all

There is AllTokenStream for indexing and AllTermQuery for querying, but I
guess AllTermsQueryParser is missing somehow, or I am missing something?

In other words in my query I need to instruct ES to use AllTermQuery and
for that I understand I need AllTermQueryParser, but there is no such thing.

My questions are:

  1. Is there any reason why such query parser is not implemented?
  2. What would be the best way to implement such query parser?
  3. Is there any better way to implement this use case?

Thanks,
Milan

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

I do not see the relationship between the DelimitedPayloadTokeFilter and
the AllTermQuery except for payloads. The lucene.all package contains
classes used for the all field.
http://www.elasticsearch.org/guide/reference/mapping/all-field/ The all
field uses the standard query parser.

The AllFieldQuery is the only elasticsearch query that uses payloads. I
often thought about how to incorporate payloads with elasticsearch, which I
think is difficult because what would the JSON representation look like?

You can create a plugin to create a new query/queryparser in ElasticSearch,
although you cannot add it to the existing QueryBuilders static imports.

Cheers,

Ivan

On Wed, Jun 12, 2013 at 5:15 AM, Milan Agatonovic <
milan.agatonovic@gmail.com> wrote:

Hi,
I am looking for the way how to use token payloads in ES, pretty much same
way as:
in DelimitedPayloadTokenFilter, described
at http://searchhub.org/2009/08/05/getting-started-with-payloads/

I see in the ES codebase almost everything I think I need.
package is: package org.elasticsearch.common.lucene.all

There is AllTokenStream for indexing and AllTermQuery for querying, but I
guess AllTermsQueryParser is missing somehow, or I am missing something?

In other words in my query I need to instruct ES to use AllTermQuery and
for that I understand I need AllTermQueryParser, but there is no such thing.

My questions are:

  1. Is there any reason why such query parser is not implemented?
  2. What would be the best way to implement such query parser?
  3. Is there any better way to implement this use case?

Thanks,
Milan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hi Ivan,
thanks for the reply.
Yes, I am certainly misled by the fact that AllTermQuery is the only query
in ES that is using payloads.
And by reading very sparse info on this forum (e.g
herehttp://elasticsearch-users.115913.n3.nabble.com/custom-similarity-setting-does-not-work-with-version-0-20-2-td4029500.html)
I was trying to find the way how to add Lucene's PayloadTermQuery into ES,
but I couldn't find the way apart from adding it to the source code. I
could not find any plugin that creates Custom Query Parser and registers it
to ES.
Is there any?

Then I read thishttps://groups.google.com/forum/?fromgroups#!topic/elasticsearch/GXUxysJ8BO4
where
it was said that you can basically use AllTermQuery to make your boosts in
payload work.
Also when I compare PayloadTermQuery and AllTermQuery they look pretty similar,
so I thought if I can "tell" ES to use AllTermQuery when I need overall
score to be calculated with the values in my payload, I'll do fine.

From the indexing perspective I think I have everything. Custom token
filter and analyzer work through analyze API. I also created custom
similarity (as per posts above) but currently I do not know how to use it.
For example both Lucene's PayloadTermQuery and AllTermQuery have their own
embedded extension of SpanScorer to scorePayload.

Any help would be much appreciated.
Thanks,
MIlan

On Thu, Jun 13, 2013 at 11:22 PM, Ivan Brusic ivan@brusic.com wrote:

I do not see the relationship between the DelimitedPayloadTokeFilter and
the AllTermQuery except for payloads. The lucene.all package contains
classes used for the all field.
http://www.elasticsearch.org/guide/reference/mapping/all-field/ The all
field uses the standard query parser.

The AllFieldQuery is the only elasticsearch query that uses payloads. I
often thought about how to incorporate payloads with elasticsearch, which I
think is difficult because what would the JSON representation look like?

You can create a plugin to create a new query/queryparser in
ElasticSearch, although you cannot add it to the existing QueryBuilders
static imports.

Cheers,

Ivan

On Wed, Jun 12, 2013 at 5:15 AM, Milan Agatonovic <
milan.agatonovic@gmail.com> wrote:

Hi,
I am looking for the way how to use token payloads in ES, pretty much
same way as:
in DelimitedPayloadTokenFilter, described
at http://searchhub.org/2009/08/05/getting-started-with-payloads/

I see in the ES codebase almost everything I think I need.
package is: package org.elasticsearch.common.lucene.all

There is AllTokenStream for indexing and AllTermQuery for querying, but I
guess AllTermsQueryParser is missing somehow, or I am missing something?

In other words in my query I need to instruct ES to use AllTermQuery and
for that I understand I need AllTermQueryParser, but there is no such thing.

My questions are:

  1. Is there any reason why such query parser is not implemented?
  2. What would be the best way to implement such query parser?
  3. Is there any better way to implement this use case?

Thanks,
Milan

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Clandestino, did you have any resolution for this? If so, would be deeply
grateful for any insights/solutions you can provide.

Regards,
devarajaswami

On Friday, June 14, 2013 at 1:33:12 AM UTC-7, clandestino_bgd wrote:

Hi Ivan,
thanks for the reply.
Yes, I am certainly misled by the fact that AllTermQuery is the only
query in ES that is using payloads.
And by reading very sparse info on this forum (e.g here
http://elasticsearch-users.115913.n3.nabble.com/custom-similarity-setting-does-not-work-with-version-0-20-2-td4029500.html)
I was trying to find the way how to add Lucene's PayloadTermQuery into ES,
but I couldn't find the way apart from adding it to the source code. I
could not find any plugin that creates Custom Query Parser and registers it
to ES.
Is there any?

Then I read this
https://groups.google.com/forum/?fromgroups#!topic/elasticsearch/GXUxysJ8BO4 where
it was said that you can basically use AllTermQuery to make your boosts in
payload work.
Also when I compare PayloadTermQuery and AllTermQuery they look pretty similar,
so I thought if I can "tell" ES to use AllTermQuery when I need overall
score to be calculated with the values in my payload, I'll do fine.

From the indexing perspective I think I have everything. Custom token
filter and analyzer work through analyze API. I also created custom
similarity (as per posts above) but currently I do not know how to use it.
For example both Lucene's PayloadTermQuery and AllTermQuery have their
own embedded extension of SpanScorer to scorePayload.

Any help would be much appreciated.
Thanks,
MIlan

On Thu, Jun 13, 2013 at 11:22 PM, Ivan Brusic <iv...@brusic.com
<javascript:>> wrote:

I do not see the relationship between the DelimitedPayloadTokeFilter and
the AllTermQuery except for payloads. The lucene.all package contains
classes used for the all field.
http://www.elasticsearch.org/guide/reference/mapping/all-field/ The all
field uses the standard query parser.

The AllFieldQuery is the only elasticsearch query that uses payloads. I
often thought about how to incorporate payloads with elasticsearch, which I
think is difficult because what would the JSON representation look like?

You can create a plugin to create a new query/queryparser in
ElasticSearch, although you cannot add it to the existing QueryBuilders
static imports.

Cheers,

Ivan

On Wed, Jun 12, 2013 at 5:15 AM, Milan Agatonovic <milan.ag...@gmail.com
<javascript:>> wrote:

Hi,
I am looking for the way how to use token payloads in ES, pretty much
same way as:
in DelimitedPayloadTokenFilter, described
at http://searchhub.org/2009/08/05/getting-started-with-payloads/

I see in the ES codebase almost everything I think I need.
package is: package org.elasticsearch.common.lucene.all

There is AllTokenStream for indexing and AllTermQuery for querying, but
I guess AllTermsQueryParser is missing somehow, or I am missing something?

In other words in my query I need to instruct ES to use AllTermQuery and
for that I understand I need AllTermQueryParser, but there is no such thing.

My questions are:

  1. Is there any reason why such query parser is not implemented?
  2. What would be the best way to implement such query parser?
  3. Is there any better way to implement this use case?

Thanks,
Milan

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/cd86ed25-3655-431c-9b66-7cc70ef262bb%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.