# Specify metadata per word/term in a string **URL:** \ **Category:** Elasticsearch\ **Created:** [April 30, 2014, 11:06am UTC](https://discuss.elastic.co/t/specify-metadata-per-word-term-in-a-string/17278 "2014-04-30T11:06:57Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Neeraj\_Makam](https://avatars.discourse-cdn.com/v4/letter/n/94ad74/32.png) [@Neeraj\_Makam](https://discuss.elastic.co/u/Neeraj_Makam)\ **Post date:** [April 30, 2014, 11:06am UTC](https://discuss.elastic.co/t/specify-metadata-per-word-term-in-a-string/17278/1 "2014-04-30T11:06:57Z")
Hi Given a text,say "hello elastic search world", is there a way i can associate a field or some metadata per word in the text on which i can later query? for eg: give code number to each word, and should be able to search like text = "hello" AND code = "25" i.e return all "hello" words which have 25 in their code metadata. -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![vineeth\_mohan\_2](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/vineeth_mohan_2/32/747_2.png) [@vineeth\_mohan\_2](https://discuss.elastic.co/u/vineeth_mohan_2)\ **Post date:** [April 30, 2014, 11:55am UTC](https://discuss.elastic.co/t/specify-metadata-per-word-term-in-a-string/17278/2 "2014-04-30T11:55:54Z")
Hello Neeraj , First of all you cant return a "hello" from Elasticsearch. Elasticsearch works on feed level basis. Which means if you want to search hello , you will get the feed with the text "hello elasticsearch search world" but not just hello. Only way I can think of create a different document for each word. So a document would look like - { "word" : "hello", "code" : 25 } here , you can get it worked. If you want to retrieve the text also , give it as follows - { "text" : "hello from Elasticsearch" , "words" : [ { "word" : "hello" , "count" : 25 } , { "word" : "from" , "count" : 22} ] } WHERE words field is nested type. Thanks Vineeth On Wed, Apr 30, 2014 at 4:36 PM, Neeraj Makam [neeraj23m92@gmail.com](mailto:neeraj23m92@gmail.com) wrote: > Hi > > Given a text,say "hello Elasticsearch world", > is there a way i can associate a field or some metadata per word in the > text on which i can later query? > for eg: give code number to each word, and should be able to search like > text = "hello" AND code = "25" > i.e return all "hello" words which have 25 in their code metadata. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nwzNDRVkkxJ8qczgzFAacatg-tEQYaAkrk%2BeHRRcqthA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nwzNDRVkkxJ8qczgzFAacatg-tEQYaAkrk%2BeHRRcqthA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Neeraj\_Makam](https://avatars.discourse-cdn.com/v4/letter/n/94ad74/32.png) [@Neeraj\_Makam](https://discuss.elastic.co/u/Neeraj_Makam)\ **Post date:** [April 30, 2014, 12:06pm UTC](https://discuss.elastic.co/t/specify-metadata-per-word-term-in-a-string/17278/3 "2014-04-30T12:06:31Z")
hi, thanks for the answer. But that would stop me from enabling highlighting for such queries. Can you think of any workaround for highlighting also to work? -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/ace6896e-bbd5-48d5-a4fa-bd9d4e6d7a0f%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/ace6896e-bbd5-48d5-a4fa-bd9d4e6d7a0f%40googlegroups.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![jprante](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jprante/32/44941_2.png) [@jprante](https://discuss.elastic.co/u/jprante)\ **Post date:** [April 30, 2014, 12:20pm UTC](https://discuss.elastic.co/t/specify-metadata-per-word-term-in-a-string/17278/4 "2014-04-30T12:20:36Z")
There is a related feature that is called payloads for terms. In Elasticsearch you can assign payload to terms, e.g. numbers for custom scoring. See also > > > It would be great to be able so have a mapping field which stores payloads with …terms and be able to use the payloads to boost the score of the document. > > In my particular use case, I have documents which are tagged by users and after running through filters and algorithms we can determine which tags are most likely useful and which are likely spam. We'd like to pass that information on to the search index so that we can boost the documents we think are most appropriate to the search terms. In this case the boost is known at indexing time and applies to the terms themselves and not to the field or the documents. > > This is something that's been possible with Lucene for quite awhile and which Solr had partial support for, but never fully implemented out of the box. (See for example http://wiki.apache.org/solr/Payloads, http://searchhub.org/2009/08/05/getting-started-with-payloads/, http://hnagtech.wordpress.com/2013/04/19/using-payloads-with-solr-4-x/). > > Ideally, it would be best to pass the payload in as a separate JSON field value in the document. The Solr tokenizer for payloads (DelimitedPayloadTokenFilterFactory) uses a delimiter, but I've found this to be problematic when dealing with user generated terms. In addition, it would be best to have the payload value somehow available in scripting so the payloads can be indexed once and then the scoring algorithms tweaked as necessary to get the right scores. > > > Below is a minimal example which you can run to see how it works in principle. D…ocumentation lists all options. > > At this point I would first like to make sure that the general concept is OK. After that I will add complete javadoc etc. > > closes #3772 > > Minimal example: > > \`\`\` > DELETE paytest > > PUT paytest > { > "mappings": { > "test": { > "properties": { > "text": { > "index\_analyzer": "fulltext\_analyzer", > "type": "string" > } > } > } > }, > "settings": { > "analysis": { > "analyzer": { > "fulltext\_analyzer": { > "filter": \[ > "my\_delimited\_payload\_filter" > \], > "tokenizer": "whitespace", > "type": "custom" > } > }, > "filter": { > "my\_delimited\_payload\_filter": { > "delimiter": "+", > "encoding": "float", > "type": "delimited\_payload\_filter" > } > } > }, > "index": { > "number\_of\_replicas": 0, > "number\_of\_shards": 1 > } > } > } > > > POST paytest/test/1 > { > "text": "the+1 quick+2 brown+3 fox+4 is quick+10" > } > > POST paytest/test/2 > { > "text": "the+1 quick+2 red+3 fox+4" > } > > POST paytest/\_refresh > > \#get the total term frequency for "quick" > > POST paytest/\_search > { > "script\_fields": { > "ttf": { > "script": "\_index\[\\"text\\"\]\[\\"quick\\"\].ttf()" > } > } > } > > \# get the term frequencies > POST paytest/\_search > { > "script\_fields": { > "freq": { > "script": "\_index\[\\"text\\"\]\[\\"quick\\"\].freq()" > } > } > } > POST paytest/test/2/\_termvector > \# get the payloads which are floats in this case > POST paytest/\_search > { > "script\_fields": { > "payloads": { > "script": "term = \_index\[\\"text\\"\].get(\\"red\\",\_PAYLOADS);payloads = \[\]; for(pos : term){payloads.add(pos.payloadAsFloat(-1));} return payloads;" > } > } > } > > \#compute very simple score: just use the term frequency > POST paytest/\_search > { > "script\_fields": { > "tv": { > "script": "\_index\[\\"text\\"\]\[\\"quick\\"\].freq()" > } > }, > "query": { > "function\_score": { > "functions": \[ > { > "script\_score": { > "script": "\_index\[\\"text\\"\]\[\\"quick\\"\].freq()" > } > } > \] > } > } > } > > \`\`\` > > and so on. see documentation for all options. > > Note that \`\_index\` was called \`\_shard\` before. See #4584 It uses DelimitedPayloadTokenFilter [http://lucene.apache.org/core/4\_7\_0/analyzers-common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html](http://lucene.apache.org/core/4_7_0/analyzers-common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html) Jörg On Wed, Apr 30, 2014 at 1:06 PM, Neeraj Makam [neeraj23m92@gmail.com](mailto:neeraj23m92@gmail.com) wrote: > Hi > > Given a text,say "hello Elasticsearch world", > is there a way i can associate a field or some metadata per word in the > text on which i can later query? > for eg: give code number to each word, and should be able to search like > text = "hello" AND code = "25" > i.e return all "hello" words which have 25 in their code metadata. > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com](https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com)[https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEV56vGmxgKceY5dcSkEsJwEwrmtVXona-MH298YOExnQ%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEV56vGmxgKceY5dcSkEsJwEwrmtVXona-MH298YOExnQ%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:32am UTC](https://discuss.elastic.co/t/specify-metadata-per-word-term-in-a-string/17278/5 "2017-07-06T01:32:29Z")