# How to index a word efficiently? **URL:** \ **Category:** Elasticsearch\ **Created:** [February 19, 2011, 2:32pm UTC](https://discuss.elastic.co/t/how-to-index-a-word-efficiently/3950 "2011-02-19T14:32:37Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![thinking23487](https://avatars.discourse-cdn.com/v4/letter/t/85e7bf/32.png) [@thinking23487](https://discuss.elastic.co/u/thinking23487)\ **Post date:** [February 19, 2011, 2:32pm UTC](https://discuss.elastic.co/t/how-to-index-a-word-efficiently/3950/1 "2011-02-19T14:32:37Z")
hi all i have a list of 120 mio. domain names (at around 2gb plain text) and the challenge is to be able to search those domains by wildcards as i can see the problem boils down to being able to do a wildcard search on a word to be honest im fairly new to nosql/document oriented db's and thus my first try was using postgresql and like cause its easy (altough i knew it wouldnt kinda work in sense of performance) a query can take at around 10 minutes or longer from my search it seems that elasticsearch provides wildcard search out of the box and scaling would also come in handy - thats why ill give it a try my first test with just inserting the domain names and query with wildcards resulted in nearly the same result as postgresql the next idea i would have is to split every domain into small chunks and ignore wildcards this means i insert an entry like 'domain:elasticsearch, chunks:el la as st ti ic cs se ea ar rc ch ela las ast .....' and chunks would be like a document i want to search and thus a search for '_elastic_' would result in striping the wildcards and only search for 'elastic' what would you suggest? could elasticsearch be the right tool for this? and if yes, any ideas oo optimization/indexing would be great. another point im not sure of is, how many servers would provide good performance and as a reference how many entries could one server handle (in this case)? replication is another topic im facing later after i gained the basic experience but to give some numbers, i tought about being able to get a count(\*) of a query for '_elastic_' in about 100 to 200 ms retrieving lets say the first 100 entries should be in nearly the same time i also know that providing facts on those few numbers i gave is hard and thus im only interested in experienced opinions or maybe what i should expect to not be fast as you may see im very unexperienced in full text search and document oriented work thx for your suggestions ---
**Author:** ![Rich\_Kroll](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/rich_kroll/32/3229_2.png) [@Rich\_Kroll](https://discuss.elastic.co/u/Rich_Kroll)\ **Post date:** [February 22, 2011, 1:05am UTC](https://discuss.elastic.co/t/how-to-index-a-word-efficiently/3950/2 "2011-02-22T01:05:46Z")
Elasticsearch allows for configuring the number of shards and replicas, which directly affects search performance. Considering the map/reduce fashion that ES queries the index, I would think you could achieve good performance against your dataset. ES breaks your documents down in the index, so I don't think you will get any benefit from attempting to do it yourself. Another option may be to write a custom analyizer, which creates tokens on characters. Then you would not need the wildcards. On Feb 21, 2011 6:47 PM, "JonathanD" [prometheus\_\_0@hotmail.com](mailto:prometheus__0@hotmail.com) wrote: ---
**Author:** ![Barsk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/barsk/32/13734_2.png) [@Barsk](https://discuss.elastic.co/u/Barsk)\ **Post date:** [February 22, 2011, 9:09am UTC](https://discuss.elastic.co/t/how-to-index-a-word-efficiently/3950/3 "2011-02-22T09:09:29Z")
``` Hi, As a newbie to Elastic Search myself I should maybe refrain from answering. But I do recognize the confusion going from a SQL environment to elasticsearch. There is a lot to grasp. And I have just begun myself. However, when it comes to indexing the domain names I would suggest using the standard analyzer that breaks down string on the dots, lowercases and removes stopwords. E.g. www.YAHOO.com = www yahoo com The standard analyzer is using the default set of english stopwords though (stopwords = words to NOT index). And you dont't want that in this case as every word is significant in this context. But you might want to skip the top level domains so why not consider those as stopwords. Something like this (you have to fill in the complete list of top level domains though). And maybe you can skip www too. This example is in YAML format. ``` ``` index : analysis : analyzer : myDomainAnalyzer: type : standard stopwords: [www,com,net,biz,gov,org,se,no,da,fi] ``` ``` Another approach is to use a pattern analyzer to skip the first host name part of the domain name, i.e[www.YAHOO.com](http://www.YAHOO.com)= yahoo.com ``` ``` index : analysis : analyzer : myDomainAnalyzer: type : pattern pattern: [\w+\.\w+$] ``` ``` A third approach is to use a keyword analyzer to get the whole domain name as is and then use wildcards to search in the string. If you want to also lowercase you can configure a custom filter setup: ``` ``` index : analysis : analyzer : myDomainAnalyzer : type: custom tokenizer: keyword filter: [lowercase] ``` ``` There is also the choice to store the index in a multi_field setup where each field is analyzed according to different analyzer setups. Then you can search with different approaches all-in-one and maybe combine the results afterwards (that specific area is beyond me ATM). Well, I am not sure this anwsers your questions, but this is what I can provide as a getting-started guide. /Kristian JonathanD skrev 2011-02-19 15:32: 

```

hi all

i have a list of 120 mio. domain names (at around 2gb plain text) and the  
challenge is to be able to search those domains  
by wildcards  
as i can see the problem boils down to being able to do a wildcard search on  
a word  
to be honest im fairly new to nosql/document oriented db's and thus my first  
try  
was using postgresql and like cause its easy (altough i knew it wouldnt  
kinda work in sense of performance)  
a query can take at around 10 minutes or longer  
from my search it seems that elasticsearch provides wildcard search out of  
the box and scaling would also come in handy - thats why ill give it a try

my first test with just inserting the domain names and query with wildcards  
resulted in nearly the same result as postgresql  
the next idea i would have is to split every domain into small chunks and  
ignore wildcards  
this means i insert an entry like 'domain:elasticsearch, chunks:el la as st  
ti ic cs se ea ar rc ch ela las ast .....' and chunks would be like a  
document i want to search and thus a search for '_elastic_' would result in  
striping the wildcards and only search for 'elastic'

what would you suggest?  
could elasticsearch be the right tool for this? and if yes, any ideas oo  
optimization/indexing would be great. another point im not sure of is, how  
many servers would provide good performance and as a reference how many  
entries could one server handle (in this case)? replication is another topic  
im facing later after i gained the basic experience  
but to give some numbers, i tought about being able to get a count(\*) of a  
query for '_elastic_' in about 100 to 200 ms  
retrieving lets say the first 100 entries should be in nearly the same time  
i also know that providing facts on those few numbers i gave is hard and  
thus im only interested in experienced opinions or maybe what i should  
expect to not be fast

as you may see im very unexperienced in full text search and document  
oriented work  
thx for your suggestions

```
-- 
Med vänlig hälsning
Kristian Jörg

Devo IT AB
Tel: 054 - 22 14 58, 0709 - 15 83 42
E-post:[kristian.jorg@devo.se](mailto:kristian.jorg@devo.se)Webb:[http://www.devo.se](http://www.devo.se)
```

---




**Author:** ![Karussell1](https://avatars.discourse-cdn.com/v4/letter/k/50afbb/32.png) [@Karussell1](https://discuss.elastic.co/u/Karussell1)\
**Post date:** [February 22, 2011, 12:20pm UTC](https://discuss.elastic.co/t/how-to-index-a-word-efficiently/3950/4 "2011-02-22T12:20:20Z")




Yes, sharding is necessary (12 shards?).

Also keep mind you'll enough for every shard and your index needs to  
be optimized after indexing.

After successfully making this fast you should try a NGramFilter for  
the wildcard thing.

Regards,  
Peter.

On 22 Feb., 02:05, Rich Kroll [kroll.r...@gmail.com](mailto:kroll.r...@gmail.com) wrote:

> Elasticsearch allows for configuring the number of shards and replicas,  
> which directly affects search performance. Considering the map/reduce  
> fashion that ES queries the index, I would think you could achieve good  
> performance against your dataset. ES breaks your documents down in the  
> index, so I don't think you will get any benefit from attempting to do it  
> yourself. Another option may be to write a custom analyizer, which creates  
> tokens on characters. Then you would not need the wildcards.  
> On Feb 21, 2011 6:47 PM, "JonathanD" [prometheus...@hotmail.com](mailto:prometheus...@hotmail.com) wrote:

---




**Author:** ![otisg](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/otisg/32/492_2.png) [@otisg](https://discuss.elastic.co/u/otisg)\
**Post date:** [February 22, 2011, 7:53pm UTC](https://discuss.elastic.co/t/how-to-index-a-word-efficiently/3950/5 "2011-02-22T19:53:51Z")




Hi,

You likely want to do the following:

- Make sure you "flip" hostname pieces ([foo.bar.com](http://foo.bar.com) becomes  
com.bar.foo) before tokenizing
- Index these pieces as multiple shingled tokens (first token is  
"com", second one is "com.bar", third one is "com.bar.foo").

The above to let you get all "com" domains, all "com.bar" subdomains,  
etc. -- without using wildcards! (but at the expense of slightly  
larger index)

- Potentially also "edge-ngram" the pieces if you want to be able to  
make prefix queries (with trailing wildcards, e.g. "com.ba\*" if you  
need to fine all ba\*.com domains.

## Otis

Sematext :: [http://sematext.com/](http://sematext.com/) :: Solr - Lucene - Elastic Search  
Lucene ecosystem search :: [http://search-lucene.com/](http://search-lucene.com/)

On Feb 19, 9:32 am, JonathanD [prometheus...@hotmail.com](mailto:prometheus...@hotmail.com) wrote:

> hi all
> 
> i have a list of 120 mio. domain names (at around 2gb plain text) and the  
> challenge is to be able to search those domains  
> by wildcards  
> as i can see the problem boils down to being able to do a wildcard search on  
> a word  
> to be honest im fairly new to nosql/document oriented db's and thus my first  
> try  
> was using postgresql and like cause its easy (altough i knew it wouldnt  
> kinda work in sense of performance)  
> a query can take at around 10 minutes or longer  
> from my search it seems that elasticsearch provides wildcard search out of  
> the box and scaling would also come in handy - thats why ill give it a try
> 
> my first test with just inserting the domain names and query with wildcards  
> resulted in nearly the same result as postgresql  
> the next idea i would have is to split every domain into small chunks and  
> ignore wildcards  
> this means i insert an entry like 'domain:elasticsearch, chunks:el la as st  
> ti ic cs se ea ar rc ch ela las ast .....' and chunks would be like a  
> document i want to search and thus a search for '_elastic_' would result in  
> striping the wildcards and only search for 'elastic'
> 
> what would you suggest?  
> could elasticsearch be the right tool for this? and if yes, any ideas oo  
> optimization/indexing would be great. another point im not sure of is, how  
> many servers would provide good performance and as a reference how many  
> entries could one server handle (in this case)? replication is another topic  
> im facing later after i gained the basic experience  
> but to give some numbers, i tought about being able to get a count(\*) of a  
> query for '_elastic_' in about 100 to 200 ms  
> retrieving lets say the first 100 entries should be in nearly the same time  
> i also know that providing facts on those few numbers i gave is hard and  
> thus im only interested in experienced opinions or maybe what i should  
> expect to not be fast
> 
> ## as you may see im very unexperienced in full text search and document oriented work thx for your suggestions
> 
> View this message in context:[http://elasticsearch-users.115913.n3.nabble.com/how-to-index-a-word-e](http://elasticsearch-users.115913.n3.nabble.com/how-to-index-a-word-e)...  
> Sent from the Elasticsearch Users mailing list archive at [Nabble.com](http://Nabble.com).

---




**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 4:11am UTC](https://discuss.elastic.co/t/how-to-index-a-word-efficiently/3950/6 "2017-07-06T04:11:41Z")