Are tokens produced by an analyzer == terms?


(Nikita Tovstoles) #1

Reference says that an Analyzer takes a piece of text, tokenizes it using
a Tokenizer, and then mutates resulting list of tokens using specified
Token Filters. Thus, if I understand correctly, the output of an analyzer
is some set of tokens. in the context of indexing a Document, are those
tokens == Terms? In other words are terms 'Tokens' and 'Terms' synonyms?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8590cbb5-911a-46bb-8ed1-a5485cb4f6df%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.


(Clinton Gormley) #2

Yes. Tokens and terms are used as synonyms, although officially there is a
difference:
http://nlp.stanford.edu/IR-book/html/htmledition/tokenization-1.html

But for our purposes, wherever you read "token" think "term"

On 12 March 2014 20:40, Nikita Tovstoles nikita.tovstoles@gmail.com wrote:

Reference says that an Analyzer takes a piece of text, tokenizes it using
a Tokenizer, and then mutates resulting list of tokens using specified
Token Filters. Thus, if I understand correctly, the output of an analyzer
is some set of tokens. in the context of indexing a Document, are those
tokens == Terms? In other words are terms 'Tokens' and 'Terms' synonyms?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/8590cbb5-911a-46bb-8ed1-a5485cb4f6df%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/8590cbb5-911a-46bb-8ed1-a5485cb4f6df%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPt3XKSV0T2Op5%2BCkTyPzMWw%3Dbdvx6rELZobkH0s2k_T7HR-Lw%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.


(system) #3