Common terms query with a mix of phrases & single word terms


(Brent Wheeldon) #1

Hi there,

We're trying to get up and running with elasticsearch and are stuck getting
common search terms with a mix of single words and phrases. My pair just
tweeted about it and Simon said he may be able to help
(https://twitter.com/s1m0nw/status/398938278587400192).

Help?

Thanks!

Brent and Danielle

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(simonw-2) #2

Hey,

can you explain what you mean by a mix of single words and phrase? It would
be great to get more insight into what you are trying to do. I assume you
mean PhraseQuery but that might be tricky. If you could use shingles as
well there might be some hope here :slight_smile:

Please elaborate a bit and / or provide an example?

simon

On Friday, November 8, 2013 11:39:03 PM UTC+1, Brent Wheeldon wrote:

Hi there,

We're trying to get up and running with elasticsearch and are stuck
getting common search terms with a mix of single words and phrases. My pair
just tweeted about it and Simon said he may be able to help (
https://twitter.com/s1m0nw/status/398938278587400192).

Help?

Thanks!

Brent and Danielle

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Brent Wheeldon) #3

Hi Simon,

Sorry for the delayed reply.

The functionality we are shooting for is exact phrase searching, so
something like "bart simpson" burns would return matches for bart simpson and matches for burns (we're using OR as our default operator).
We have this working just fine using a query string query. The next piece
of functionality that we would like is to filter out stop words. While
looking into this, we found the common terms query documentation and this
seems to be a much nicer solution that traditional stop words. However,
we've spun our wheels a little trying to work out how to retain our exact
phrase match functionality, and introduce the common terms query. At this
point, we're just not sure what the best way forward is.

What is the recommended way of getting this done?

Cheers,

Brent

On 8 November 2013 18:05, simonw simon.willnauer@elasticsearch.com wrote:

Hey,

can you explain what you mean by a mix of single words and phrase? It
would be great to get more insight into what you are trying to do. I assume
you mean PhraseQuery but that might be tricky. If you could use shingles as
well there might be some hope here :slight_smile:

Please elaborate a bit and / or provide an example?

simon

On Friday, November 8, 2013 11:39:03 PM UTC+1, Brent Wheeldon wrote:

Hi there,

We're trying to get up and running with elasticsearch and are stuck
getting common search terms with a mix of single words and phrases. My pair
just tweeted about it and Simon said he may be able to help (
https://twitter.com/s1m0nw/status/398938278587400192).

Help?

Thanks!

Brent and Danielle

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(simonw-2) #4

so if I understand this correctly you want to use phrase matches but remove
the stopwords our of the phrase? this doesn't make much sense to me since
it won't really help performance wise. I am afraid the common terms
functionality is currently not supported for for phrases at all. I might
not understand what you are trying to do yet so feel free to enlighten me
:slight_smile: I'd be interested in what the reason for exact phrases is do you want to
have better precision? Is it a hard requirement and do you have to go
through the query parser?

simon

On Monday, November 11, 2013 3:22:30 PM UTC+1, Brent Wheeldon wrote:

Hi Simon,

Sorry for the delayed reply.

The functionality we are shooting for is exact phrase searching, so
something like "bart simpson" burns would return matches for bart simpson and matches for burns (we're using OR as our default operator).
We have this working just fine using a query string query. The next piece
of functionality that we would like is to filter out stop words. While
looking into this, we found the common terms query documentation and this
seems to be a much nicer solution that traditional stop words. However,
we've spun our wheels a little trying to work out how to retain our exact
phrase match functionality, and introduce the common terms query. At this
point, we're just not sure what the best way forward is.

What is the recommended way of getting this done?

Cheers,

Brent

On 8 November 2013 18:05, simonw <simon.w...@elasticsearch.com<javascript:>

wrote:

Hey,

can you explain what you mean by a mix of single words and phrase? It
would be great to get more insight into what you are trying to do. I assume
you mean PhraseQuery but that might be tricky. If you could use shingles as
well there might be some hope here :slight_smile:

Please elaborate a bit and / or provide an example?

simon

On Friday, November 8, 2013 11:39:03 PM UTC+1, Brent Wheeldon wrote:

Hi there,

We're trying to get up and running with elasticsearch and are stuck
getting common search terms with a mix of single words and phrases. My pair
just tweeted about it and Simon said he may be able to help (
https://twitter.com/s1m0nw/status/398938278587400192).

Help?

Thanks!

Brent and Danielle

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Brent Wheeldon) #5

Sorry, I'm obviously not doing a great job of describing our situation.

If we assume that the is a common word in our index, we have a model
which has field_a and field_b indexed with two records, one with foo bart simpson baz in field_a, and one with burns in field_b. When we
search for "bart simpson" the burns we want to see both records in the
result set.

Is that clearer?

On 12 November 2013 14:52, simonw simon.willnauer@elasticsearch.com wrote:

so if I understand this correctly you want to use phrase matches but
remove the stopwords our of the phrase? this doesn't make much sense to me
since it won't really help performance wise. I am afraid the common terms
functionality is currently not supported for for phrases at all. I might
not understand what you are trying to do yet so feel free to enlighten me
:slight_smile: I'd be interested in what the reason for exact phrases is do you want to
have better precision? Is it a hard requirement and do you have to go
through the query parser?

simon

On Monday, November 11, 2013 3:22:30 PM UTC+1, Brent Wheeldon wrote:

Hi Simon,

Sorry for the delayed reply.

The functionality we are shooting for is exact phrase searching, so
something like "bart simpson" burns would return matches for bart simpson and matches for burns (we're using OR as our default operator).
We have this working just fine using a query string query. The next piece
of functionality that we would like is to filter out stop words. While
looking into this, we found the common terms query documentation and this
seems to be a much nicer solution that traditional stop words. However,
we've spun our wheels a little trying to work out how to retain our exact
phrase match functionality, and introduce the common terms query. At this
point, we're just not sure what the best way forward is.

What is the recommended way of getting this done?

Cheers,

Brent

On 8 November 2013 18:05, simonw simon.w...@elasticsearch.com wrote:

Hey,

can you explain what you mean by a mix of single words and phrase? It
would be great to get more insight into what you are trying to do. I assume
you mean PhraseQuery but that might be tricky. If you could use shingles as
well there might be some hope here :slight_smile:

Please elaborate a bit and / or provide an example?

simon

On Friday, November 8, 2013 11:39:03 PM UTC+1, Brent Wheeldon wrote:

Hi there,

We're trying to get up and running with elasticsearch and are stuck
getting common search terms with a mix of single words and phrases. My pair
just tweeted about it and Simon said he may be able to help (
https://twitter.com/s1m0nw/status/398938278587400192).

Help?

Thanks!

Brent and Danielle

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(simonw-2) #6

ah I see so the problem that I see is that you can only use "phrase
operation" with the query parser which doesn't support common terms. so
either you drop the phrase and use maybe shingles to get better precision
or you filter stopwords in you analysis chain.

simon

On Tuesday, November 12, 2013 9:03:09 PM UTC+1, Brent Wheeldon wrote:

Sorry, I'm obviously not doing a great job of describing our situation.

If we assume that the is a common word in our index, we have a model
which has field_a and field_b indexed with two records, one with foo bart simpson baz in field_a, and one with burns in field_b. When we
search for "bart simpson" the burns we want to see both records in the
result set.

Is that clearer?

On 12 November 2013 14:52, simonw <simon.w...@elasticsearch.com<javascript:>

wrote:

so if I understand this correctly you want to use phrase matches but
remove the stopwords our of the phrase? this doesn't make much sense to me
since it won't really help performance wise. I am afraid the common terms
functionality is currently not supported for for phrases at all. I might
not understand what you are trying to do yet so feel free to enlighten me
:slight_smile: I'd be interested in what the reason for exact phrases is do you want to
have better precision? Is it a hard requirement and do you have to go
through the query parser?

simon

On Monday, November 11, 2013 3:22:30 PM UTC+1, Brent Wheeldon wrote:

Hi Simon,

Sorry for the delayed reply.

The functionality we are shooting for is exact phrase searching, so
something like "bart simpson" burns would return matches for bart simpson and matches for burns (we're using OR as our default operator).
We have this working just fine using a query string query. The next piece
of functionality that we would like is to filter out stop words. While
looking into this, we found the common terms query documentation and this
seems to be a much nicer solution that traditional stop words. However,
we've spun our wheels a little trying to work out how to retain our exact
phrase match functionality, and introduce the common terms query. At this
point, we're just not sure what the best way forward is.

What is the recommended way of getting this done?

Cheers,

Brent

On 8 November 2013 18:05, simonw simon.w...@elasticsearch.com wrote:

Hey,

can you explain what you mean by a mix of single words and phrase? It
would be great to get more insight into what you are trying to do. I assume
you mean PhraseQuery but that might be tricky. If you could use shingles as
well there might be some hope here :slight_smile:

Please elaborate a bit and / or provide an example?

simon

On Friday, November 8, 2013 11:39:03 PM UTC+1, Brent Wheeldon wrote:

Hi there,

We're trying to get up and running with elasticsearch and are stuck
getting common search terms with a mix of single words and phrases. My pair
just tweeted about it and Simon said he may be able to help (
https://twitter.com/s1m0nw/status/398938278587400192).

Help?

Thanks!

Brent and Danielle

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Brent Wheeldon) #7

Bummer. Thanks for the info!

On Tuesday, November 12, 2013, simonw wrote:

ah I see so the problem that I see is that you can only use "phrase
operation" with the query parser which doesn't support common terms. so
either you drop the phrase and use maybe shingles to get better precision
or you filter stopwords in you analysis chain.

simon

On Tuesday, November 12, 2013 9:03:09 PM UTC+1, Brent Wheeldon wrote:

Sorry, I'm obviously not doing a great job of describing our situation.

If we assume that the is a common word in our index, we have a model
which has field_a and field_b indexed with two records, one with foo bart simpson baz in field_a, and one with burns in field_b. When we
search for "bart simpson" the burns we want to see both records in the
result set.

Is that clearer?

On 12 November 2013 14:52, simonw simon.w...@elasticsearch.com wrote:

so if I understand this correctly you want to use phrase matches but
remove the stopwords our of the phrase? this doesn't make much sense to me
since it won't really help performance wise. I am afraid the common terms
functionality is currently not supported for for phrases at all. I might
not understand what you are trying to do yet so feel free to enlighten me
:slight_smile: I'd be interested in what the reason for exact phrases is do you want to
have better precision? Is it a hard requirement and do you have to go
through the query parser?

simon

On Monday, November 11, 2013 3:22:30 PM UTC+1, Brent Wheeldon wrote:

Hi Simon,

Sorry for the delayed reply.

The functionality we are shooting for is exact phrase searching, so
something like "bart simpson" burns would return matches for bart simpson and matches for burns (we're using OR as our default operator).
We have this working just fine using a query string query. The next piece
of functionality that we would like is to filter out stop words. While
looking into this, we found the common terms query documentation and this
seems to be a much nicer solution that traditional stop words. However,
we've spun our wheels a little trying to work out how to retain our exact
phrase match functionality, and introduce the common terms query. At this
point, we're just not sure what the best way forward is.

What is the recommended way of getting this done?

Cheers,

Brent

On 8 November 2013 18:05, simonw simon.w...@elasticsearch.com wrote:

Hey,

can you explain what you mean by a mix of single words and phrase? It
would be great to get more insight into what you are trying to do. I assume
you mean PhraseQuery but that might be tricky. If you could use shingles as
well there might be some hope here :slight_smile:

Please elaborate a bit and / or provide an example?

simon

On Friday, November 8, 2013 11:39:03 PM UTC+1, Brent Wheeldon wrote:

Hi there,

We're trying to get up and running with elasticsearch and are stuck
getting common search terms with a mix of single words and phrases. My pair
just tweeted about it and Simon said he may be able to help (
https://twitter.com/s1m0nw/status/398938278587400192).

Help?

Thanks!

Brent and Danielle

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/to
pic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.

For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit https://groups.google.com/d/
topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/zznOYCYWnUM/unsubscribe.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com <javascript:_e({}, 'cvml',
'elasticsearch%2Bunsubscribe@googlegroups.com');>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #8