# Custom tokenizer or analyzer? **URL:** \ **Category:** Elasticsearch\ **Created:** [May 5, 2012, 5:50pm UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570 "2012-05-05T17:50:02Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [May 5, 2012, 5:50pm UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570/1 "2012-05-05T17:50:02Z")
I have a local property in my documents. It can be a simple language code or a language/country combo. i.e. { locale: 'en' } or { locale: 'en\_GB' } My search truth looks like this: ``` document locale en en_GB en_US ``` s e en yes yes yes a r en\_GB yes yes yes c h es no no no I would like to tokenize these values in this way: 'en' =\> 'en' 'en\_GB' =\> 'en', 'en\_gb' This is intended to allow me to find the correct document. I am not sure how to integrate such a tokenization into ES. Any hints? Of course this is just matching, not sorting at this moment. For sorting, I would want the most specific match to score higher. ---
**Author:** ![Clinton\_Gormley](https://avatars.discourse-cdn.com/v4/letter/c/50afbb/32.png) [@Clinton\_Gormley](https://discuss.elastic.co/u/Clinton_Gormley)\ **Post date:** [May 7, 2012, 9:02am UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570/2 "2012-05-07T09:02:19Z")
Hi Jim On Sat, 2012-05-05 at 10:50 -0700, James Cook wrote: > I have a local property in my documents. It can be a simple language > code or a language/country combo. > { locale: 'en' } or { locale: 'en\_GB' } > I would like to tokenize these values in this way: > 'en' =\> 'en' > 'en\_GB' =\> 'en', 'en\_gb' I think the easiest way to achieve this is by using a multi-field: > > Of course this is just matching, not sorting at this moment. For > sorting, I would want the most specific match to score higher. In the gist above, I show one way that you could sort your results. clint ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [May 7, 2012, 1:01pm UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570/3 "2012-05-07T13:01:39Z")
That is so cool, thanks. I gotta be honest, after using ES for two years now and building three projects that use ES as their datastore, I know very little about the Lucene side (mapping and querying) of things. I hope a book is forthcoming, as this is really hard and somewhat unapproachable for those of us without the Lucene experience. An example like yours would be an awesome cookbook entry. Don't know why we haven't made an ES cookbook section on the website. Thanks again. On Saturday, May 5, 2012 1:50:02 PM UTC-4, James Cook wrote: > I have a local property in my documents. It can be a simple language code > or a language/country combo. > > i.e. > > { locale: 'en' } or { locale: 'en\_GB' } > > My search truth looks like this: > > ``` > document locale > en en_GB en_US > > ``` > > s > e en yes yes yes > a > r en\_GB yes yes yes > c > h es no no no > > I would like to tokenize these values in this way: > 'en' =\> 'en' > 'en\_GB' =\> 'en', 'en\_gb' > > This is intended to allow me to find the correct document. I am not sure > how to integrate such a tokenization into ES. Any hints? > > Of course this is just matching, not sorting at this moment. For sorting, > I would want the most specific match to score higher. ---
**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\ **Post date:** [May 7, 2012, 6:15pm UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570/4 "2012-05-07T18:15:00Z")
Well, if you lack knowledge on the Lucene side, then there already is a great book on the subject: [Lucene in Action, Second Edition](http://www.manning.com/hatcher3/) The analysis/tokenization chapters are very good. Some of the book is already outdated, but the analysis part remains the same. Lucene does have a documentation problem. Very little on the website and the contrib libs have almost no documentation. The mailing list however is good and not very chatty (much less than this one), so don't be afraid to sign up. An ES book would be great, but I see the system as still being a moving system. Plus, it Shay were to write a book, that means less time working on ES! -- Ivan On Mon, May 7, 2012 at 6:01 AM, James Cook [jcook@pykl.com](mailto:jcook@pykl.com) wrote: > That is so cool, thanks. > > I gotta be honest, after using ES for two years now and building three > projects that use ES as their datastore, I know very little about the Lucene > side (mapping and querying) of things. I hope a book is forthcoming, as this > is really hard and somewhat unapproachable for those of us without the > Lucene experience. An example like yours would be an awesome cookbook entry. > Don't know why we haven't made an ES cookbook section on the website. > > Thanks again. > > On Saturday, May 5, 2012 1:50:02 PM UTC-4, James Cook wrote: > > > I have a local property in my documents. It can be a simple language code > > or a language/country combo. > > > > i.e. > > > > { locale: 'en' } or { locale: 'en\_GB' } > > > > My search truth looks like this: > > > > ``` > > document locale > > en en_GB en_US > > > > ``` > > > > s > > e en yes yes yes > > a > > r en\_GB yes yes yes > > c > > h es no no no > > > > I would like to tokenize these values in this way: > > 'en' =\> 'en' > > 'en\_GB' =\> 'en', 'en\_gb' > > > > This is intended to allow me to find the correct document. I am not sure > > how to integrate such a tokenization into ES. Any hints? > > > > Of course this is just matching, not sorting at this moment. For sorting, > > I would want the most specific match to score higher. ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [May 7, 2012, 7:44pm UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570/5 "2012-05-07T19:44:58Z")
I know we have toyed with an online cookbook for a number of years now. Perhaps I can find some time to make a pull request towards this effort. I need to sleep two hours less a day. Edison must of been on to something. > **[Yahoo | Mail, Weather, Search, Politics, News, Finance, Sports & Videos](https://www.yahoo.com/)** > > Latest news coverage, email, free stock quotes, live scores and video are just the beginning. Discover more every day at Yahoo! -- jim On Monday, May 7, 2012 2:15:00 PM UTC-4, Ivan Brusic wrote: > Well, if you lack knowledge on the Lucene side, then there already is > a great book on the subject: [Lucene in Action, Second Edition](http://www.manning.com/hatcher3/) > > The analysis/tokenization chapters are very good. Some of the book is > already outdated, but the analysis part remains the same. Lucene does > have a documentation problem. Very little on the website and the > contrib libs have almost no documentation. The mailing list however is > good and not very chatty (much less than this one), so don't be afraid > to sign up. > > An ES book would be great, but I see the system as still being a > moving system. Plus, it Shay were to write a book, that means less > time working on ES! > > -- > Ivan > > On Mon, May 7, 2012 at 6:01 AM, James Cook wrote: > > > That is so cool, thanks. > > > > I gotta be honest, after using ES for two years now and building three > > projects that use ES as their datastore, I know very little about the > > Lucene > > side (mapping and querying) of things. I hope a book is forthcoming, as > > this > > is really hard and somewhat unapproachable for those of us without the > > Lucene experience. An example like yours would be an awesome cookbook > > entry. > > Don't know why we haven't made an ES cookbook section on the website. > > > > Thanks again. > > > > On Saturday, May 5, 2012 1:50:02 PM UTC-4, James Cook wrote: > > > > > I have a local property in my documents. It can be a simple language > > > code > > > or a language/country combo. > > > > > > i.e. > > > > > > { locale: 'en' } or { locale: 'en\_GB' } > > > > > > My search truth looks like this: > > > > > > ``` > > > document locale > > > en en_GB en_US > > > > > > ``` > > > > > > s > > > e en yes yes yes > > > a > > > r en\_GB yes yes yes > > > c > > > h es no no no > > > > > > I would like to tokenize these values in this way: > > > 'en' =\> 'en' > > > 'en\_GB' =\> 'en', 'en\_gb' > > > > > > This is intended to allow me to find the correct document. I am not > > > sure > > > how to integrate such a tokenization into ES. Any hints? > > > > > > Of course this is just matching, not sorting at this moment. For > > > sorting, > > > I would want the most specific match to score higher. ---
**Author:** ![James\_Cook\_3](https://avatars.discourse-cdn.com/v4/letter/j/d2c977/32.png) [@James\_Cook\_3](https://discuss.elastic.co/u/James_Cook_3)\ **Post date:** [June 7, 2012, 8:42pm UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570/6 "2012-06-07T20:42:54Z")
The approach of using multifield types for a locale value works well in my use case of returning a list of matching articles in order of the most specific locale value. Assume I have a property called 'key' which uniquely identifies an article topic, and the combination of 'key' and 'locale' identifies a specific document in my index. For arguments sake, assume I have several documents with the same 'key', but different locales. The query and mapping Clinton presented works very well to return me a list of those matching documents in order of most specific locale match for a specific key. But what happens if I want a list of all documents (multiple keys) and pass a specific locale? I will get a list of all matching documents (by locale) for each key. But I really want a single match for each document key. Any thoughts on how to filter this list? On Monday, May 7, 2012 5:02:19 AM UTC-4, Clinton Gormley wrote: > Hi Jim > > On Sat, 2012-05-05 at 10:50 -0700, James Cook wrote: > > > I have a local property in my documents. It can be a simple language > > code or a language/country combo. > > > { locale: 'en' } or { locale: 'en\_GB' } > > > I would like to tokenize these values in this way: > > 'en' =\> 'en' > > 'en\_GB' =\> 'en', 'en\_gb' > > I think the easiest way to achieve this is by using a multi-field: > > [gist:2626785 ยท GitHub](https://gist.github.com/2626785) > > > Of course this is just matching, not sorting at this moment. For > > sorting, I would want the most specific match to score higher. > > In the gist above, I show one way that you could sort your results. > > clint ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:25am UTC](https://discuss.elastic.co/t/custom-tokenizer-or-analyzer/7570/7 "2017-07-06T03:25:19Z")