# Help needed with the query

**URL:** <https://discuss.elastic.co/t/help-needed-with-the-query/4878>\
**Category:** Elasticsearch\
**Created:** [July 17, 2011, 8:17pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878 "2011-07-17T20:17:38Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![mfeingold](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mfeingold](https://discuss.elastic.co/u/mfeingold)\
**Post date:** [July 17, 2011, 8:17pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/1 "2011-07-17T20:17:38Z")

</div>

Can you guys help me understand how should I approach my problem. Here  
is what I need to do:  
As a user types information about a member I need to provide a list of  
suggestions based on the provided information. The information may or  
may not include the following: member first/last name, his member id  
and date of birth.

It is my understanding that I need to index my membership info (~100M  
records) so that I can use edit distance and metaphone against first  
name/ last name. I also would like to use a list of synonyms for the  
first name ('Bob' vs 'Robert'). It also makes sense to use edit  
distance against member id - to account for typing errors. Date of  
birth I would like to use as a filter - if it is there I only show  
members matching it.  
Also to further narrow the list I would only include members who live  
within certain distance of the service location.

All of this is just thinking aloud (in writing?). I am not sure how  
this should (or can it) be translated into ES configuration /query.  
Any help?

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 18, 2011, 11:28am UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/2 "2011-07-18T11:28:02Z")

</div>

Hi Michael

> As a user types information about a member I need to provide a list of  
> suggestions based on the ...: member first/last name, his member id  
> and date of birth.

> It is my understanding that I need to index my membership info (~100M  
> records) so that I can use edit distance and metaphone against first  
> name/ last name. I also would like to use a list of synonyms for the  
> first name ('Bob' vs 'Robert').

Yes - preparing your data correctly is essential.

> It also makes sense to use edit  
> distance against member id - to account for typing errors.

Do you really think this is the case? If somebody is typing a member ID,  
then I think they should see JUST the associated user. Otherwise, if  
all I have is the member ID, I type that, and it shows me 20 different  
users, how do I know which is the one I want? I'll ignore this  
requirement.

> Date of birth I would like to use as a filter - if it is there I only  
> show members matching it.

> Also to further narrow the list I would only include members who live  
> within certain distance of the service location.

I apologise in advance - this email is long, but is well worth reading  
(and I should probably turn it into a tutorial, as this question is  
asked often):

OK, so there are two phases here:

1. preparing your data, and
2. searching

## PREPARING YOUR DATA:

First, let's decide how each field needs to be indexed, then we can look  
at what analyzers we need to provide.

- first name / last name:

- member id

- birthday

- location

So we have a list of custom analyzers we need to define:

- full\_name:

- partial\_name:

- name\_metaphone:

Here is the command to create the index with the above analyzers and  
mapping: [Create index for partial matching of names in ElasticSearch · GitHub](https://gist.github.com/1088986)

It is quite long, but just goes through the process listed above. If you  
look at each block, it's actually quite simple.

NOTES:

1. For first\_name/last\_name, I am using multi-fields. The "main"  
sub-field has the same name as the top level, so that if I refer  
to "first\_name" it automatically references "first\_name.first\_name"

2. In the partial name fields, I am using index\_analyzer and  
search\_analyzer.

So, run the commands in the gist above, and then you can experiment with  
searches.

You can see what tokens each analyzer produces with the analyze API.  
Try these queries:

curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=full\_name](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=full_name)'  
curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=partial\_name](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=partial_name)'  
curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=name\_metaphone](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=name_metaphone)'

and to check that the ascii folding is working, try 'sánchez' (but URL encoded):

curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=sánchez&analyzer=partial\_name](http://127.0.0.1:9200/test/_analyze?pretty=1&text=s%C3%A1nchez&analyzer=partial_name)'

## SEARCHING YOUR DATA

Let's get rid of the easy stuff first:

birthday:

```
    If your user enters a birthday, then you want to filter the
    results to only include members with a matching birthday:
    
       { term: { birthday: '1970-10-24' }}

```

location:

```
    Use a geo_distance filter to find results within 100km of
    London:
    
       { geo_distance: { 
               distance: "100km",
               location: [51.50853, -0.12574]
       }}

```

member\_id:

```
 This can use a simple text query:

 { text: { member_id: "abc-1234" }}

```

OK - now the more interesting stuff: first name and last name.

The logic we want to use here is:

Show me any name whose first or last name field matches  
completely or partially, but consider full word matches  
to be more relevant than partial matches or metaphone  
matches

We're going to combine these queries using the 'bool' query. The  
difference between the 'bool' query and the 'dismax' query is that the  
'bool' query combines the \_score/relevance of each matching clause,  
while the 'dismax' query chooses the highest \_score from the matching  
clauses.

{ bool:  
{ should: [  
{ text: { "first\_name": "rob" }}, # full name  
{ text: { "first\_name.partial": "rob" }} # partial match  
{ text: { "first\_name.metaphone": "rob"}} # metaphone  
]  
}}

This will find all docs that match any of the above clauses.

The \_score of each matching clause is combined, so a doc which matches  
all 3 clauses will rank higher than a doc that matches just one clause,  
so we already have some ranking here.

But lets say that we wanted a full word match to be significantly more  
relevant than the other two. We can change that clause to:

{ text: { "first\_name": {  
query: "rob",  
boost: 1  
}}}

Of course, we need to include the same 3 clauses for "last\_name" as  
well.

Now, to turn these into a query that we can pass to the Search API:

All search queries must be wrapped in a top-level { query: {....}}  
element, which will contain one of 3 possibilities:

1. just the bool query  
{ query: { bool: {...}}

2. just one or more filters  
{ query: { constant\_score: { filter: {....} }}

3. the bool query combined with one or more filters  
{ query: { filtered: { query: {bool: ...}, filter: {.....} }}}

This was long, but I hope it was worth it.

If anything isn't clear, please ask, and I can improve this and turn it  
into a tutorial.

clint

---

<div class="post-metadata">

**Author:** ![mfeingold](https://avatars.discourse-cdn.com/v4/letter/m/ecb155/32.png) [@mfeingold](https://discuss.elastic.co/u/mfeingold)\
**Post date:** [July 18, 2011, 4:16pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/3 "2011-07-18T16:16:53Z")

</div>

Hi Clinton:

Thanks for a quick and detailed response.

To clarify a few points:

1. The synonyms - I only need the synonyms on the first name, I  
think. I do not imagine it to be of much use for last names. I am not  
sure if dropping synonyms for the last name would have any impact in  
terms of performance or necessary disk space. Based on your templates  
I hope I understand how to do this.
2. Names - one of the problems I foresee is stemming from the fact  
that I want a single input string with all the search parameters  
(except geoloc). The problem is to tell apart the first name from the  
last name. Additional complication here is that a name (both first and  
last) can consist of several words. My hope was that I can build  
indexes/queries in such a way so that I can throw both names as a  
single string at the query leaving it to the ES to figure out which  
one is which
3. Edge ngrams - I would like to limit wild card search to first let  
us say 6 chars, assuming that more than that implies exact match. my  
clumsy experiments with the ES made me think that if ngrams are in the  
play, they have to be all the way through the max length of the field,  
otherwise the search misses exact matches. I was doing something  
wrong, I hope
4. Member ID - it is alphanumeric. I still think that some degree of  
fuzziness can help here. The idea is to let the user type anything he  
knows and provide autosuggest as he types. So if he knows the ID -  
great it should be an immediate hit, but if he mistyped it and also  
provided last name - it still can let me to make a pretty well  
educated guess. So why do not do it? I think though that the edit  
distance allowed here should be minimal.
5. Geo - I am curious about the performance of this. Does it really  
does sqrt from sum of squares? Does it mean that during this process  
it actually loops through all documents to find the matching ones? The  
reason I am asking is that I do not insist on the circle - I can get  
away with a square, I mean instead of real geo distance here - I can  
use a range on both longitude and latitude. Would it be faster?

On Jul 18, 6:28 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote:

> Hi Michael
> 
> > As a user types information about a member I need to provide a list of  
> > suggestions based on the ...: member first/last name, his member id  
> > and date of birth.
> 
> > It is my understanding that I need to index my membership info (~100M  
> > records) so that I can use edit distance and metaphone against first  
> > name/ last name. I also would like to use a list of synonyms for the  
> > first name ('Bob' vs 'Robert').
> 
> Yes - preparing your data correctly is essential.
> 
> > It also makes sense to use edit  
> > distance against member id - to account for typing errors.
> 
> Do you really think this is the case? If somebody is typing a member ID,  
> then I think they should see JUST the associated user. Otherwise, if  
> all I have is the member ID, I type that, and it shows me 20 different  
> users, how do I know which is the one I want? I'll ignore this  
> requirement.
> 
> > Date of birth I would like to use as a filter - if it is there I only  
> > show members matching it.  
> > Also to further narrow the list I would only include members who live  
> > within certain distance of the service location.
> 
> I apologise in advance - this email is long, but is well worth reading  
> (and I should probably turn it into a tutorial, as this question is  
> asked often):
> 
> OK, so there are two phases here:
> 
> 1. preparing your data, and
> 2. searching
> 
> ## PREPARING YOUR DATA:
> 
> First, let's decide how each field needs to be indexed, then we can look  
> at what analyzers we need to provide.
> 
> - first name / last name:
> 
> - member id
> 
> - birthday
> 
> - location
> 
> So we have a list of custom analyzers we need to define:
> 
> - full\_name:
> 
> - partial\_name:
> 
> - name\_metaphone:
> 
> Here is the command to create the index with the above analyzers and  
> mapping:[Create index for partial matching of names in ElasticSearch · GitHub](https://gist.github.com/1088986)
> 
> It is quite long, but just goes through the process listed above. If you  
> look at each block, it's actually quite simple.
> 
> NOTES:
> 
> 1. For first\_name/last\_name, I am using multi-fields. The "main"  
> sub-field has the same name as the top level, so that if I refer  
> to "first\_name" it automatically references "first\_name.first\_name"
> 
> 2. In the partial name fields, I am using index\_analyzer and  
> search\_analyzer.
> 
> So, run the commands in the gist above, and then you can experiment with  
> searches.
> 
> You can see what tokens each analyzer produces with the analyze API.  
> Try these queries:
> 
> curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=full\_n](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=full_n)...  
> curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=partia](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=partia)...  
> curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=name\_m](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=name_m)...
> 
> and to check that the ascii folding is working, try 'sánchez' (but URL encoded):
> 
> curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=sánchez&analyz](http://127.0.0.1:9200/test/_analyze?pretty=1&text=s%C3%A1nchez&analyz)...
> 
> ## SEARCHING YOUR DATA
> 
> Let's get rid of the easy stuff first:
> 
> birthday:
> 
> ```
> If your user enters a birthday, then you want to filter the
> results to only include members with a matching birthday:
> 
> { term: { birthday: '1970-10-24' }}
> 
> ```
> 
> location:
> 
> ```
> Use a geo_distance filter to find results within 100km of
> London:
> 
> { geo_distance: {
> distance: "100km",
> location: [51.50853, -0.12574]
> }}
> 
> ```
> 
> member\_id:
> 
> ```
> This can use a simple text query:
> 
> { text: { member_id: "abc-1234" }}
> 
> ```
> 
> OK - now the more interesting stuff: first name and last name.
> 
> The logic we want to use here is:
> 
> Show me any name whose first or last name field matches  
> completely or partially, but consider full word matches  
> to be more relevant than partial matches or metaphone  
> matches
> 
> We're going to combine these queries using the 'bool' query. The  
> difference between the 'bool' query and the 'dismax' query is that the  
> 'bool' query combines the \_score/relevance of each matching clause,  
> while the 'dismax' query chooses the highest \_score from the matching  
> clauses.
> 
> { bool:  
> { should: [  
> { text: { "first\_name": "rob" }}, # full name  
> { text: { "first\_name.partial": "rob" }} # partial match  
> { text: { "first\_name.metaphone": "rob"}} # metaphone  
> ]
> 
> }}
> 
> This will find all docs that match any of the above clauses.
> 
> The \_score of each matching clause is combined, so a doc which matches  
> all 3 clauses will rank higher than a doc that matches just one clause,  
> so we already have some ranking here.
> 
> But lets say that we wanted a full word match to be significantly more  
> relevant than the other two. We can change that clause to:
> 
> { text: { "first\_name": {  
> query: "rob",  
> boost: 1  
> }}}
> 
> Of course, we need to include the same 3 clauses for "last\_name" as  
> well.
> 
> Now, to turn these into a query that we can pass to the Search API:
> 
> All search queries must be wrapped in a top-level { query: {....}}  
> element, which will contain one of 3 possibilities:
> 
> 1. just the bool query  
> { query: { bool: {...}}
> 
> 2. just one or more filters  
> { query: { constant\_score: { filter: {....} }}
> 
> 3. the bool query combined with one or more filters  
> { query: { filtered: { query: {bool: ...}, filter: {.....} }}}
> 
> This was long, but I hope it was worth it.
> 
> If anything isn't clear, please ask, and I can improve this and turn it  
> into a tutorial.
> 
> clint

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 18, 2011, 4:44pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/4 "2011-07-18T16:44:42Z")

</div>

Hi Michael

> 1. The synonyms - I only need the synonyms on the first name, I  
> think. I do not imagine it to be of much use for last names. I am not  
> sure if dropping synonyms for the last name would have any impact in  
> terms of performance or necessary disk space. Based on your templates  
> I hope I understand how to do this.

That's fine, just define one custom analyzer with synonyms, and use that  
for first names, and another without synonyms, for the last name.

> 1. Names - one of the problems I foresee is stemming from the fact  
> that I want a single input string with all the search parameters  
> (except geoloc). The problem is to tell apart the first name from the  
> last name. Additional complication here is that a name (both first and  
> last) can consist of several words. My hope was that I can build  
> indexes/queries in such a way so that I can throw both names as a  
> single string at the query leaving it to the ES to figure out which  
> one is which

In my example, "rob smith" looks for "rob OR smith", and you're running  
that against first name and last name, so you can use the same search  
string. It will find 'rob' in first name and 'smith' in last name.

> 1. Edge ngrams - I would like to limit wild card search to first let  
> us say 6 chars, assuming that more than that implies exact match. my  
> clumsy experiments with the ES made me think that if ngrams are in the  
> play, they have to be all the way through the max length of the field,  
> otherwise the search misses exact matches. I was doing something  
> wrong, I hope

You can limit the ngrams, but there is no real reason to do so. Also,  
you have the full word version of the field which it will match against.  
In the bool query I'm using 'should' (which is like 'or') so not all  
fields need to match.

> 1. Member ID - it is alphanumeric. I still think that some degree of  
> fuzziness can help here. The idea is to let the user type anything he  
> knows and provide autosuggest as he types. So if he knows the ID -  
> great it should be an immediate hit, but if he mistyped it and also  
> provided last name - it still can let me to make a pretty well  
> educated guess. So why do not do it? I think though that the edit  
> distance allowed here should be minimal.

That's probably fine - with the text query you can use the 'fuzziness'  
parameter,

> 1. Geo - I am curious about the performance of this. Does it really  
> does sqrt from sum of squares? Does it mean that during this process  
> it actually loops through all documents to find the matching ones? The  
> reason I am asking is that I do not insist on the circle - I can get  
> away with a square, I mean instead of real geo distance here - I can  
> use a range on both longitude and latitude. Would it be faster?

geo\_distance is fast. no idea how it works internally, but no worries  
there

clint

> On Jul 18, 6:28 am, Clinton Gormley [clin...@iannounce.co.uk](mailto:clin...@iannounce.co.uk) wrote:
> 
> > Hi Michael
> > 
> > > As a user types information about a member I need to provide a list of  
> > > suggestions based on the ...: member first/last name, his member id  
> > > and date of birth.
> > 
> > > It is my understanding that I need to index my membership info (~100M  
> > > records) so that I can use edit distance and metaphone against first  
> > > name/ last name. I also would like to use a list of synonyms for the  
> > > first name ('Bob' vs 'Robert').
> > 
> > Yes - preparing your data correctly is essential.
> > 
> > > It also makes sense to use edit  
> > > distance against member id - to account for typing errors.
> > 
> > Do you really think this is the case? If somebody is typing a member ID,  
> > then I think they should see JUST the associated user. Otherwise, if  
> > all I have is the member ID, I type that, and it shows me 20 different  
> > users, how do I know which is the one I want? I'll ignore this  
> > requirement.
> > 
> > > Date of birth I would like to use as a filter - if it is there I only  
> > > show members matching it.  
> > > Also to further narrow the list I would only include members who live  
> > > within certain distance of the service location.
> > 
> > I apologise in advance - this email is long, but is well worth reading  
> > (and I should probably turn it into a tutorial, as this question is  
> > asked often):
> > 
> > OK, so there are two phases here:
> > 
> > 1. preparing your data, and
> > 2. searching
> > 
> > ## PREPARING YOUR DATA:
> > 
> > First, let's decide how each field needs to be indexed, then we can look  
> > at what analyzers we need to provide.
> > 
> > - first name / last name:
> > 
> > - member id
> > 
> > - birthday
> > 
> > - location
> > 
> > So we have a list of custom analyzers we need to define:
> > 
> > - full\_name:
> > 
> > - partial\_name:
> > 
> > - name\_metaphone:
> > 
> > Here is the command to create the index with the above analyzers and  
> > mapping:[Create index for partial matching of names in ElasticSearch · GitHub](https://gist.github.com/1088986)
> > 
> > It is quite long, but just goes through the process listed above. If you  
> > look at each block, it's actually quite simple.
> > 
> > NOTES:
> > 
> > 1. For first\_name/last\_name, I am using multi-fields. The "main"  
> > sub-field has the same name as the top level, so that if I refer  
> > to "first\_name" it automatically references "first\_name.first\_name"
> > 
> > 2. In the partial name fields, I am using index\_analyzer and  
> > search\_analyzer.
> > 
> > So, run the commands in the gist above, and then you can experiment with  
> > searches.
> > 
> > You can see what tokens each analyzer produces with the analyze API.  
> > Try these queries:
> > 
> > curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=full\_n](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=full_n)...  
> > curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=partia](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=partia)...  
> > curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=name\_m](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=name_m)...
> > 
> > and to check that the ascii folding is working, try 'sánchez' (but URL encoded):
> > 
> > curl -XGET '[http://127.0.0.1:9200/test/\_analyze?pretty=1&text=sánchez&analyz](http://127.0.0.1:9200/test/_analyze?pretty=1&text=s%C3%A1nchez&analyz)...
> > 
> > ## SEARCHING YOUR DATA
> > 
> > Let's get rid of the easy stuff first:
> > 
> > birthday:
> > 
> > ```
> > If your user enters a birthday, then you want to filter the
> > results to only include members with a matching birthday:
> > 
> > { term: { birthday: '1970-10-24' }}
> > 
> > ```
> > 
> > location:
> > 
> > ```
> > Use a geo_distance filter to find results within 100km of
> > London:
> > 
> > { geo_distance: {
> > distance: "100km",
> > location: [51.50853, -0.12574]
> > }}
> > 
> > ```
> > 
> > member\_id:
> > 
> > ```
> > This can use a simple text query:
> > 
> > { text: { member_id: "abc-1234" }}
> > 
> > ```
> > 
> > OK - now the more interesting stuff: first name and last name.
> > 
> > The logic we want to use here is:
> > 
> > Show me any name whose first or last name field matches  
> > completely or partially, but consider full word matches  
> > to be more relevant than partial matches or metaphone  
> > matches
> > 
> > We're going to combine these queries using the 'bool' query. The  
> > difference between the 'bool' query and the 'dismax' query is that the  
> > 'bool' query combines the \_score/relevance of each matching clause,  
> > while the 'dismax' query chooses the highest \_score from the matching  
> > clauses.
> > 
> > { bool:  
> > { should: [  
> > { text: { "first\_name": "rob" }}, # full name  
> > { text: { "first\_name.partial": "rob" }} # partial match  
> > { text: { "first\_name.metaphone": "rob"}} # metaphone  
> > ]
> > 
> > }}
> > 
> > This will find all docs that match any of the above clauses.
> > 
> > The \_score of each matching clause is combined, so a doc which matches  
> > all 3 clauses will rank higher than a doc that matches just one clause,  
> > so we already have some ranking here.
> > 
> > But lets say that we wanted a full word match to be significantly more  
> > relevant than the other two. We can change that clause to:
> > 
> > { text: { "first\_name": {  
> > query: "rob",  
> > boost: 1  
> > }}}
> > 
> > Of course, we need to include the same 3 clauses for "last\_name" as  
> > well.
> > 
> > Now, to turn these into a query that we can pass to the Search API:
> > 
> > All search queries must be wrapped in a top-level { query: {....}}  
> > element, which will contain one of 3 possibilities:
> > 
> > 1. just the bool query  
> > { query: { bool: {...}}
> > 
> > 2. just one or more filters  
> > { query: { constant\_score: { filter: {....} }}
> > 
> > 3. the bool query combined with one or more filters  
> > { query: { filtered: { query: {bool: ...}, filter: {.....} }}}
> > 
> > This was long, but I hope it was worth it.
> > 
> > If anything isn't clear, please ask, and I can improve this and turn it  
> > into a tutorial.
> > 
> > clint

--  
Web Announcements Limited is a company registered in England and Wales,  
with company number 05608868, with registered address at 10 Arvon Road,  
London, N5 1PR.

---

<div class="post-metadata">

**Author:** ![kimchy](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kimchy/32/44952_2.png) [@kimchy](https://discuss.elastic.co/u/kimchy)\
**Post date:** [July 18, 2011, 5:29pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/5 "2011-07-18T17:29:01Z")

</div>

I suggest you start with mapping. Start with a simple one, where you have  
mappings set for different elements (first name, last name), with custom  
analyzers that you define that use what you want. You might need to use  
multi\_field mapping type if you want to have several analyzers applied to  
the same field.

On Sun, Jul 17, 2011 at 11:17 PM, Michael Feingold [mfeingold@hill30.com](mailto:mfeingold@hill30.com)wrote:

> Can you guys help me understand how should I approach my problem. Here  
> is what I need to do:  
> As a user types information about a member I need to provide a list of  
> suggestions based on the provided information. The information may or  
> may not include the following: member first/last name, his member id  
> and date of birth.
> 
> It is my understanding that I need to index my membership info (~100M  
> records) so that I can use edit distance and metaphone against first  
> name/ last name. I also would like to use a list of synonyms for the  
> first name ('Bob' vs 'Robert'). It also makes sense to use edit  
> distance against member id - to account for typing errors. Date of  
> birth I would like to use as a filter - if it is there I only show  
> members matching it.  
> Also to further narrow the list I would only include members who live  
> within certain distance of the service location.
> 
> All of this is just thinking aloud (in writing?). I am not sure how  
> this should (or can it) be translated into ES configuration /query.  
> Any help?

---

<div class="post-metadata">

**Author:** ![lalit\_mishra](https://avatars.discourse-cdn.com/v4/letter/l/b77776/32.png) [@lalit\_mishra](https://discuss.elastic.co/u/lalit_mishra)\
**Post date:** [July 19, 2011, 3:15pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/6 "2011-07-19T15:15:43Z")

</div>

Hi Clinton,  
In place of ngram can I use prefixQuery to serve the purpose is there any  
advantage of using ngram tokenizer?

Below configuration is an example  
{  
"tweet" : {  
"properties" : {  
"shortName" : {  
"type" : "multi\_field",  
"fields" : {  
"name" : {"type" : "string", "index" : "analyzed"},  
"untouched" : {"type" : "string", "index" :  
"not\_analyzed"}  
}  
}  
}  
}  
}

query name.untouched for exact search using textPhrase  
and prefix query for partial search

Please let me know if you thing otherwise

Thanks,  
Lalit.

On Mon, Jul 18, 2011 at 4:58 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote:

> Hi Michael
> 
> > As a user types information about a member I need to provide a list of  
> > suggestions based on the ...: member first/last name, his member id  
> > and date of birth.
> 
> > It is my understanding that I need to index my membership info (~100M  
> > records) so that I can use edit distance and metaphone against first  
> > name/ last name. I also would like to use a list of synonyms for the  
> > first name ('Bob' vs 'Robert').
> 
> Yes - preparing your data correctly is essential.
> 
> > It also makes sense to use edit  
> > distance against member id - to account for typing errors.
> 
> Do you really think this is the case? If somebody is typing a member ID,  
> then I think they should see JUST the associated user. Otherwise, if  
> all I have is the member ID, I type that, and it shows me 20 different  
> users, how do I know which is the one I want? I'll ignore this  
> requirement.
> 
> > Date of birth I would like to use as a filter - if it is there I only  
> > show members matching it.
> 
> > Also to further narrow the list I would only include members who live  
> > within certain distance of the service location.
> 
> I apologise in advance - this email is long, but is well worth reading  
> (and I should probably turn it into a tutorial, as this question is  
> asked often):
> 
> OK, so there are two phases here:
> 
> 1. preparing your data, and
> 2. searching
> 
> ## PREPARING YOUR DATA:
> 
> First, let's decide how each field needs to be indexed, then we can look  
> at what analyzers we need to provide.
> 
> - first name / last name:
> 
> - these are string fields
> 
> - we want to use synonyms (eg Robert vs Bob)
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/synonym-tokenfilter.html)
> 
> - we want to include metaphones
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/phonetic-tokenfilter.html)
> 
> - we want to find 'clinton' if the user types 'clin' (auto-complete), so  
> we'll use  
> edge ngrams:  
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/edgengram-tokenfilter.html)
> 
> - and let's also use ascii-folding to allow 'éëè' to match 'e'
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/asciifolding-tokenfilter.html)
> 
> - we want to do 3 types of matches:
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/mapping/multi-field-type.html)
> 
> - member id
> - you didn't specify if this is an numeric or alphanumeric, so I'll  
> just assume alphanumeric, possibly with punctuation, eg  
> "ABC-1234"
> 
> - let's say that we want to tokenize this as "abc","1234", so we'll  
> use the "simple" analyzer
> 
> [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/index-modules/analysis/simple-analyzer.html)
> 
> - birthday
> 
> - location
> 
> So we have a list of custom analyzers we need to define:
> 
> - full\_name:
> 
> - partial\_name:
> 
> - name\_metaphone:
> 
> Here is the command to create the index with the above analyzers and  
> mapping: [Create index for partial matching of names in ElasticSearch · GitHub](https://gist.github.com/1088986)
> 
> It is quite long, but just goes through the process listed above. If you  
> look at each block, it's actually quite simple.
> 
> NOTES:
> 
> 1. For first\_name/last\_name, I am using multi-fields. The "main"  
> sub-field has the same name as the top level, so that if I refer  
> to "first\_name" it automatically references "first\_name.first\_name"
> 
> So in effect, I have "first\_name" and "first\_name.partial" and  
> "first\_name.metaphone"
> 
> 1. In the partial name fields, I am using index\_analyzer and  
> search\_analyzer.
> 
> Normally, you want your data and search terms to use the same  
> analyzer - this ensures that you are searching for the same  
> terms that are actually stored in ES. For example, in the  
> first\_name.metaphone field, I just specify an 'analyzer'  
> (which sets both the search\_analyzer and index\_analyzer to  
> the same value)
> 
> However, for the partial field, we want them to be different. If we  
> store the name "Clinton", we want to be able to use auto-complete  
> for search terms like 'clin' (ie partial matches). So at index time,  
> we tokenize clinton as c,cl,cli,clin,clint,clinto,clinton
> 
> However, when we search, we don't want 'clin' to match 'cat','cliff'  
> etc. So we DON'T want to use the ngram tokenizer on search terms.
> 
> So, run the commands in the gist above, and then you can experiment with  
> searches.
> 
> You can see what tokens each analyzer produces with the analyze API.  
> Try these queries:
> 
> curl -XGET '  
> [http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=full\_name](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=full_name)'  
> curl -XGET '  
> [http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=partial\_name](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=partial_name)  
> '  
> curl -XGET '  
> [http://127.0.0.1:9200/test/\_analyze?pretty=1&text=rob&analyzer=name\_metaphone](http://127.0.0.1:9200/test/_analyze?pretty=1&text=rob&analyzer=name_metaphone)  
> '
> 
> and to check that the ascii folding is working, try 'sánchez' (but URL  
> encoded):
> 
> curl -XGET '  
> [http://127.0.0.1:9200/test/\_analyze?pretty=1&text=sánchez&analyzer=partial\_name](http://127.0.0.1:9200/test/_analyze?pretty=1&text=s%C3%A1nchez&analyzer=partial_name)  
> '
> 
> ## SEARCHING YOUR DATA
> 
> Let's get rid of the easy stuff first:
> 
> birthday:
> 
> ```
> If your user enters a birthday, then you want to filter the
> results to only include members with a matching birthday:
> 
> { term: { birthday: '1970-10-24' }}
> 
> ```
> 
> location:
> 
> ```
> Use a geo_distance filter to find results within 100km of
> London:
> 
> { geo_distance: {
> distance: "100km",
> location: [51.50853, -0.12574]
> }}
> 
> ```
> 
> member\_id:
> 
> ```
> This can use a simple text query:
> 
> { text: { member_id: "abc-1234" }}
> 
> ```
> 
> OK - now the more interesting stuff: first name and last name.
> 
> The logic we want to use here is:
> 
> Show me any name whose first or last name field matches  
> completely or partially, but consider full word matches  
> to be more relevant than partial matches or metaphone  
> matches
> 
> We're going to combine these queries using the 'bool' query. The  
> difference between the 'bool' query and the 'dismax' query is that the  
> 'bool' query combines the \_score/relevance of each matching clause,  
> while the 'dismax' query chooses the highest \_score from the matching  
> clauses.
> 
> { bool:  
> { should: [  
> { text: { "first\_name": "rob" }}, # full name  
> { text: { "first\_name.partial": "rob" }} # partial match  
> { text: { "first\_name.metaphone": "rob"}} # metaphone  
> ]  
> }}
> 
> This will find all docs that match any of the above clauses.
> 
> The \_score of each matching clause is combined, so a doc which matches  
> all 3 clauses will rank higher than a doc that matches just one clause,  
> so we already have some ranking here.
> 
> But lets say that we wanted a full word match to be significantly more  
> relevant than the other two. We can change that clause to:
> 
> { text: { "first\_name": {  
> query: "rob",  
> boost: 1  
> }}}
> 
> Of course, we need to include the same 3 clauses for "last\_name" as  
> well.
> 
> Now, to turn these into a query that we can pass to the Search API:
> 
> All search queries must be wrapped in a top-level { query: {....}}  
> element, which will contain one of 3 possibilities:
> 
> 1. just the bool query  
> { query: { bool: {...}}
> 
> [Example name query · GitHub](https://gist.github.com/1089180)
> 
> 1. just one or more filters  
> { query: { constant\_score: { filter: {....} }}
> 
> [Filter just by name and geo distance · GitHub](https://gist.github.com/1089206)
> 
> 1. the bool query combined with one or more filters  
> { query: { filtered: { query: {bool: ...}, filter: {.....} }}}
> 
> [Name query filtered by birthday and geo distance · GitHub](https://gist.github.com/1089201)
> 
> This was long, but I hope it was worth it.
> 
> If anything isn't clear, please ask, and I can improve this and turn it  
> into a tutorial.
> 
> clint

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 19, 2011, 3:23pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/7 "2011-07-19T15:23:08Z")

</div>

Hi Lalit

> In place of ngram can I use prefixQuery to serve the purpose is there  
> any advantage of using ngram tokenizer?

Performance. The prefix query is easy to use, but nowhere near as  
efficient. first it needs to find all terms which might match, then run  
queries on all of those. And, you may have too many matching terms,  
etc.

So prefix query is fine for small numbers of terms, but ngrams will  
scale

clint

---

<div class="post-metadata">

**Author:** ![lalit\_mishra](https://avatars.discourse-cdn.com/v4/letter/l/b77776/32.png) [@lalit\_mishra](https://discuss.elastic.co/u/lalit_mishra)\
**Post date:** [July 19, 2011, 4:19pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/8 "2011-07-19T16:19:56Z")

</div>

Thanks Clinton for quick response.

My knowledge with edge ngram is very less can you please put somelight over  
what actually is edge ngram ?  
Even I would like to use edge ngram

Thanks,  
Lalit.

On Tue, Jul 19, 2011 at 8:53 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote:

> Hi Lalit
> 
> > In place of ngram can I use prefixQuery to serve the purpose is there  
> > any advantage of using ngram tokenizer?
> 
> Performance. The prefix query is easy to use, but nowhere near as  
> efficient. first it needs to find all terms which might match, then run  
> queries on all of those. And, you may have too many matching terms,  
> etc.
> 
> So prefix query is fine for small numbers of terms, but ngrams will  
> scale
> 
> clint

---

<div class="post-metadata">

**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\
**Post date:** [July 19, 2011, 4:27pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/9 "2011-07-19T16:27:15Z")

</div>

> My knowledge with edge ngram is very less can you please put somelight  
> over what actually is edge ngram ?  
> Even I would like to use edge ngram

An ngram is a moving window, so an ngram of length 2 of the word "help"  
would give you "he","el","lp"

An edge-ngram is anchored to either the beginning or the end of the  
word, eg "h","he","hel","help" or (from the end) "help","elp","lp","p"

clint

> Thanks,  
> Lalit.
> 
> On Tue, Jul 19, 2011 at 8:53 PM, Clinton Gormley  
> [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote:  
> Hi Lalit
> 
> ```
> > In place of ngram can I use prefixQuery to serve the purpose
> is there
> > any advantage of using ngram tokenizer?
>     
>     
> Performance. The prefix query is easy to use, but nowhere
> near as
> efficient. first it needs to find all terms which might
> match, then run
> queries on all of those. And, you may have too many matching
> terms,
> etc.
>     
> So prefix query is fine for small numbers of terms, but ngrams
> will
> scale
>     
> clint
> 
> ```

--  
Web Announcements Limited is a company registered in England and Wales,  
with company number 05608868, with registered address at 10 Arvon Road,  
London, N5 1PR.

---

<div class="post-metadata">

**Author:** ![lalit\_mishra](https://avatars.discourse-cdn.com/v4/letter/l/b77776/32.png) [@lalit\_mishra](https://discuss.elastic.co/u/lalit_mishra)\
**Post date:** [July 19, 2011, 4:28pm UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/10 "2011-07-19T16:28:28Z")

</div>

Cool Thanks 🙂

On Tue, Jul 19, 2011 at 9:57 PM, Clinton Gormley [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk)wrote:

> > My knowledge with edge ngram is very less can you please put somelight  
> > over what actually is edge ngram ?  
> > Even I would like to use edge ngram
> 
> An ngram is a moving window, so an ngram of length 2 of the word "help"  
> would give you "he","el","lp"
> 
> An edge-ngram is anchored to either the beginning or the end of the  
> word, eg "h","he","hel","help" or (from the end) "help","elp","lp","p"
> 
> clint
> 
> > Thanks,  
> > Lalit.
> > 
> > On Tue, Jul 19, 2011 at 8:53 PM, Clinton Gormley  
> > [clinton@iannounce.co.uk](mailto:clinton@iannounce.co.uk) wrote:  
> > Hi Lalit
> > 
> > ```
> > > In place of ngram can I use prefixQuery to serve the purpose
> > is there
> > > any advantage of using ngram tokenizer?
> > 
> > Performance. The prefix query is easy to use, but nowhere
> > near as
> > efficient. first it needs to find all terms which might
> > match, then run
> > queries on all of those. And, you may have too many matching
> > terms,
> > etc.
> > 
> > So prefix query is fine for small numbers of terms, but ngrams
> > will
> > scale
> > 
> > clint
> > 
> > ```
> 
> --  
> Web Announcements Limited is a company registered in England and Wales,  
> with company number 05608868, with registered address at 10 Arvon Road,  
> London, N5 1PR.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:00am UTC](https://discuss.elastic.co/t/help-needed-with-the-query/4878/11 "2017-07-06T04:00:10Z")

</div>