# Best way to search/index the data - with and without whitespace

**URL:** <https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482>\
**Category:** Elasticsearch\
**Created:** [January 29, 2014, 7:53pm UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482 "2014-01-29T19:53:26Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![thale\_jacobs](https://avatars.discourse-cdn.com/v4/letter/t/b19c9b/32.png) [@thale\_jacobs](https://discuss.elastic.co/u/thale_jacobs)\
**Post date:** [January 29, 2014, 7:53pm UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482/1 "2014-01-29T19:53:26Z")

</div>

Hello - I am having a problem indexing and searching for words that may or  
may not contain whitespace...Below is an example

Here is how the index is created:

curl -s -XPUT 'localhost:9200/test/name/1' -d '{ "street": "Lakeshore Dr" }'  
curl -s -XPUT 'localhost:9200/test/name/2' -d '{ "street": "Sunnyshore Dr"  
}'  
curl -s -XPUT 'localhost:9200/test/name/3' -d '{ "street": "Lake View Dr" }'  
curl -s -XPUT 'localhost:9200/test/name/4' -d '{ "street": "Shore Dr" }'

If I want to query for record 1/"Lakeshore Dr", I can using the following  
query:

curl -s -XGET 'localhost:9200/test/name/\_search?pretty=true' -d '{  
"query":{  
"bool":{  
"must":[  
{  
"match":{  
"street":{  
"query":"lakeshore dr",  
"type":"phrase"  
}  
}  
}  
]  
}  
}  
}';

This returns the desired result of document id 1. But if a user searches  
for "Lake Shore Dr" (a space between Lake and Shore), it is still desired  
to return document id 1.

And the inverse of this problem is if a user searches for "Lakeview Dr"  
(but indexed as "Lake View Dr"):  
curl -s -XGET 'localhost:9200/test/name/\_search?pretty=true' -d '{  
"query":{  
"bool":{  
"must":[  
{  
"match":{  
"street":{  
"query":"lakeview dr",  
"type":"phrase"  
}  
}  
}  
]  
}  
}  
}';

The search matches to no documents. If the search is changed to a booleansearch instead of a phrase  
,  
many docs will match on "dr", but doc #3, "Lake Shore" is not necessarily  
returned as the top match.

NGrams at index time?? Ngrams at search time?? Remove whitespace at index  
time/search time??

Any suggestions would be appreciated. Thanks.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/06538a83-17d1-446c-9b27-cebf12c6fc47%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/06538a83-17d1-446c-9b27-cebf12c6fc47%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\
**Post date:** [January 29, 2014, 11:49pm UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482/2 "2014-01-29T23:49:10Z")

</div>

Thale, I would try edge ngrams (both index and search) and see how that  
works. I don't see why it wouldn't work for your 2 cases - just make your  
queries into match queries and use the "AND" operator. Good luck!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/16c67efb-0d8a-48df-a58f-4a2842c0cfda%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/16c67efb-0d8a-48df-a58f-4a2842c0cfda%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![thale\_jacobs](https://avatars.discourse-cdn.com/v4/letter/t/b19c9b/32.png) [@thale\_jacobs](https://discuss.elastic.co/u/thale_jacobs)\
**Post date:** [February 6, 2014, 1:47pm UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482/3 "2014-02-06T13:47:04Z")

</div>

Hello Binh Ly - Thanks for the replay. I thought I had read that ngram  
searching should only be used at either index time or search time, but not  
both... Is that not the case? Thanks again. Thale

On Wednesday, January 29, 2014 6:49:10 PM UTC-5, Binh Ly wrote:

> Thale, I would try edge ngrams (both index and search) and see how that  
> works. I don't see why it wouldn't work for your 2 cases - just make your  
> queries into match queries and use the "AND" operator. Good luck!

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ef6a8b2f-e291-419f-8a8b-1eefa8657d2b%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ef6a8b2f-e291-419f-8a8b-1eefa8657d2b%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\
**Post date:** [February 6, 2014, 3:16pm UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482/4 "2014-02-06T15:16:40Z")

</div>

Thale, you are correct - ngrams are usually used at index-time only, but in  
your case and requirements, you might want to experiment both index and  
seach time. I'd probably just increase the edge min ngram size to something  
reasonable like maybe 4(?) and see if that works or not.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/a86542f0-0c79-4e06-9f12-3c7200f855e9%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/a86542f0-0c79-4e06-9f12-3c7200f855e9%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![thale\_jacobs](https://avatars.discourse-cdn.com/v4/letter/t/b19c9b/32.png) [@thale\_jacobs](https://discuss.elastic.co/u/thale_jacobs)\
**Post date:** [February 6, 2014, 8:36pm UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482/5 "2014-02-06T20:36:10Z")

</div>

This is how I set up the mappings:

curl -s -XPUT 'localhost:9200/test' -d '{  
"mappings": {  
"properties": {  
"name": {  
"street": {  
"type": "string",  
"index\_analyzer": "index\_ngram",  
"search\_analyzer": "search\_ngram"  
}  
}  
}  
},  
"settings": {  
"analysis": {  
"filter": {  
"desc\_ngram": {  
"type": "edgeNGram",  
"min\_gram": 3,  
"max\_gram": 20  
}  
},  
"analyzer": {  
"index\_ngram": {  
"type": "custom",  
"tokenizer": "keyword",  
"filter": ["desc\_ngram", "lowercase"]  
},  
"search\_ngram": {  
"type": "custom",  
"tokenizer": "keyword",  
"filter": "lowercase"  
}  
}  
}  
}  
}'

This is how I built the index:

curl -s -XPUT 'localhost:9200/test/name/1' -d '{ "street": "Lakeshore Dr" }'  
curl -s -XPUT 'localhost:9200/test/name/2' -d '{ "street": "Sunnyshore Dr"  
}'  
curl -s -XPUT 'localhost:9200/test/name/3' -d '{ "street": "Lake View Dr" }'  
curl -s -XPUT 'localhost:9200/test/name/4' -d '{ "street": "Shore Dr" }'

If a user attempts to search for "Lake Shore Dr", I want to only match to  
document 1/"Lakeshore Dr"  
If a user attempts to search for "Lakeview Dr", I want to only match to  
document 3/"Lake View Dr"

Here is an example of the query that is not working correctly:

curl -s -XGET 'localhost:9200/test/\_search?pretty=true' -d '{  
"query":{  
"bool":{  
"must":[  
{  
"match":{  
"street":{  
"query":"lake shore dr",  
"type":"boolean"  
}  
}  
}  
]  
}  
}  
}';

So is the issue with how I am setting up the mappings (tokenizer?, edgegram  
vs ngrams?, size of ngrams?) or the query (I have tried things like setting  
the minimum\_should\_match, and the analyzer to use), but I have not been  
able to get the desired results.

Thanks all.

On Thursday, February 6, 2014 10:16:40 AM UTC-5, Binh Ly wrote:

> Thale, you are correct - ngrams are usually used at index-time only, but  
> in your case and requirements, you might want to experiment both index and  
> seach time. I'd probably just increase the edge min ngram size to something  
> reasonable like maybe 4(?) and see if that works or not.

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/3b7a9d63-3a08-4cfc-96ce-4b22d44cd9db%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/3b7a9d63-3a08-4cfc-96ce-4b22d44cd9db%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![Binh\_Ly](https://avatars.discourse-cdn.com/v4/letter/b/ce7236/32.png) [@Binh\_Ly](https://discuss.elastic.co/u/Binh_Ly)\
**Post date:** [February 7, 2014, 7:27pm UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482/6 "2014-02-07T19:27:56Z")

</div>

Thale,

I played with your data a little and it turns out it is more complex than I  
thought. Something like this works somewhat but may require some  
fine-tuning depending on your exact requirements. Anyway give this a try  
and see how it works (BTW I did this in ES 1.0 RC 2):

1. PUT [http://localhost:9200/test](http://localhost:9200/test)  
{  
"settings": {  
"index": {  
"number\_of\_shards": 1,  
"number\_of\_replicas": 0,  
"analysis": {  
"analyzer": {  
"en1": {  
"tokenizer": "standard",  
"filter": [  
"standard",  
"lowercase",  
"en1"  
]  
}  
},  
"filter": {  
"en1": {  
"type" : "ngram",  
"min\_gram" : 4,  
"max\_gram" : 4  
}  
}  
}  
}  
},  
"mappings": {  
"doc": {  
"properties": {  
"street": {  
"type": "string",  
"analyzer": "en1"  
}  
}  
}  
}  
}

2. POST [http://localhost:9200/test/doc/\_bulk](http://localhost:9200/test/doc/_bulk)  
{ "index": {} }  
{ "street": "Lakeshore Dr" }  
{ "index": {} }  
{ "street": "Sunnyshore Dr" }  
{ "index": {} }  
{ "street": "Lake View Dr" }  
{ "index": {} }  
{ "street": "Shore Dr" }

Query example:

GET [http://localhost:9200/test/doc/\_search](http://localhost:9200/test/doc/_search)  
{  
"query": {  
"match": {  
"street": {  
"query": "lake shore dr",  
"minimum\_should\_match": "3\<50%"  
}  
}  
}  
}

--  
You received this message because you are subscribed to the Google Groups "elasticsearch" group.  
To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com).  
To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/b3c62899-30ad-4223-9dfe-7053e2c72f72%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/b3c62899-30ad-4223-9dfe-7053e2c72f72%40googlegroups.com).  
For more options, visit [https://groups.google.com/groups/opt\_out](https://groups.google.com/groups/opt_out).

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 1:51am UTC](https://discuss.elastic.co/t/best-way-to-search-index-the-data-with-and-without-whitespace/15482/7 "2017-07-06T01:51:40Z")

</div>