Specify metadata per word/term in a string

Neeraj_Makam · April 30, 2014, 11:06am

Hi

Given a text,say "hello elastic search world",
is there a way i can associate a field or some metadata per word in the
text on which i can later query?
for eg: give code number to each word, and should be able to search like
text = "hello" AND code = "25"
i.e return all "hello" words which have 25 in their code metadata.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

vineeth_mohan_2 · April 30, 2014, 11:55am

Hello Neeraj ,

First of all you cant return a "hello" from Elasticsearch.
Elasticsearch works on feed level basis.
Which means if you want to search hello , you will get the feed with the
text "hello elasticsearch search world" but not just hello.

Only way I can think of create a different document for each word. So a
document would look like -

{
"word" : "hello",
"code" : 25
}

here , you can get it worked.

If you want to retrieve the text also , give it as follows -

{
"text" : "hello from Elasticsearch" ,
"words" : [
{ "word" : "hello" , "count" : 25 } ,
{ "word" : "from" , "count" : 22}
]
}

WHERE words field is nested type.

Thanks
Vineeth

On Wed, Apr 30, 2014 at 4:36 PM, Neeraj Makam neeraj23m92@gmail.com wrote:

Hi

Given a text,say "hello Elasticsearch world",
is there a way i can associate a field or some metadata per word in the
text on which i can later query?
for eg: give code number to each word, and should be able to search like
text = "hello" AND code = "25"
i.e return all "hello" words which have 25 in their code metadata.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGdPd5nwzNDRVkkxJ8qczgzFAacatg-tEQYaAkrk%2BeHRRcqthA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Neeraj_Makam · April 30, 2014, 12:06pm

hi, thanks for the answer.
But that would stop me from enabling highlighting for such queries. Can you
think of any workaround for highlighting also to work?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/ace6896e-bbd5-48d5-a4fa-bd9d4e6d7a0f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

jprante · April 30, 2014, 12:20pm

There is a related feature that is called payloads for terms. In
Elasticsearch you can assign payload to terms, e.g. numbers for custom
scoring.

Add support for using payloads to boost terms

opened 03:42PM - 24 Sep 13 UTC

closed 10:28AM - 02 Jan 14 UTC

bdurand

>feature v1.0.0.RC1 v0.90.10

It would be great to be able so have a mapping field which stores payloads with …terms and be able to use the payloads to boost the score of the document. In my particular use case, I have documents which are tagged by users and after running through filters and algorithms we can determine which tags are most likely useful and which are likely spam. We'd like to pass that information on to the search index so that we can boost the documents we think are most appropriate to the search terms. In this case the boost is known at indexing time and applies to the terms themselves and not to the field or the documents. This is something that's been possible with Lucene for quite awhile and which Solr had partial support for, but never fully implemented out of the box. (See for example http://wiki.apache.org/solr/Payloads, http://searchhub.org/2009/08/05/getting-started-with-payloads/, http://hnagtech.wordpress.com/2013/04/19/using-payloads-with-solr-4-x/). Ideally, it would be best to pass the payload in as a separate JSON field value in the document. The Solr tokenizer for payloads (DelimitedPayloadTokenFilterFactory) uses a delimiter, but I've found this to be problematic when dealing with user generated terms. In addition, it would be best to have the payload value somehow available in scripting so the payloads can be indexed once and then the scoring algorithms tweaked as necessary to get the right scores.

github.com/elastic/elasticsearch

make term statistics and term vectors accessible in scripts

elastic:master ← brwe:issue-3772

opened 01:27PM - 13 Nov 13 UTC

brwe

+2841 -7

Below is a minimal example which you can run to see how it works in principle. D…ocumentation lists all options. At this point I would first like to make sure that the general concept is OK. After that I will add complete javadoc etc. closes #3772 Minimal example: ``` DELETE paytest PUT paytest { "mappings": { "test": { "properties": { "text": { "index_analyzer": "fulltext_analyzer", "type": "string" } } } }, "settings": { "analysis": { "analyzer": { "fulltext_analyzer": { "filter": [ "my_delimited_payload_filter" ], "tokenizer": "whitespace", "type": "custom" } }, "filter": { "my_delimited_payload_filter": { "delimiter": "+", "encoding": "float", "type": "delimited_payload_filter" } } }, "index": { "number_of_replicas": 0, "number_of_shards": 1 } } } POST paytest/test/1 { "text": "the+1 quick+2 brown+3 fox+4 is quick+10" } POST paytest/test/2 { "text": "the+1 quick+2 red+3 fox+4" } POST paytest/_refresh #get the total term frequency for "quick" POST paytest/_search { "script_fields": { "ttf": { "script": "_index[\"text\"][\"quick\"].ttf()" } } } # get the term frequencies POST paytest/_search { "script_fields": { "freq": { "script": "_index[\"text\"][\"quick\"].freq()" } } } POST paytest/test/2/_termvector # get the payloads which are floats in this case POST paytest/_search { "script_fields": { "payloads": { "script": "term = _index[\"text\"].get(\"red\",_PAYLOADS);payloads = []; for(pos : term){payloads.add(pos.payloadAsFloat(-1));} return payloads;" } } } #compute very simple score: just use the term frequency POST paytest/_search { "script_fields": { "tv": { "script": "_index[\"text\"][\"quick\"].freq()" } }, "query": { "function_score": { "functions": [ { "script_score": { "script": "_index[\"text\"][\"quick\"].freq()" } } ] } } } ``` and so on. see documentation for all options. Note that `_index` was called `_shard` before. See #4584

It uses DelimitedPayloadTokenFilter

http://lucene.apache.org/core/4_7_0/analyzers-common/org/apache/lucene/analysis/payloads/DelimitedPayloadTokenFilter.html

Jörg

On Wed, Apr 30, 2014 at 1:06 PM, Neeraj Makam neeraj23m92@gmail.com wrote:

Hi

Given a text,say "hello Elasticsearch world",
is there a way i can associate a field or some metadata per word in the
text on which i can later query?
for eg: give code number to each word, and should be able to search like
text = "hello" AND code = "25"
i.e return all "hello" words which have 25 in their code metadata.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com https://groups.google.com/d/msgid/elasticsearch/f5f40059-e9e7-4a05-93f8-aacdff26abb0%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAKdsXoEV56vGmxgKceY5dcSkEsJwEwrmtVXona-MH298YOExnQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.