Role of analyzer,tokenizer and token filter in searching?


(Sam-4) #1

Hi

Analyzer will be applied on two levels index and search .index level
means analyzer will applied on when index is prepared,same for
tokenizer and filter and second level will applied on when doing some
search.just my curiosity i have some question to you:

  1. using both level can make any difference on result or any one will
    be enough to get required result.

for example:

if my index data will carry following string-"my name is john".
i have applied index level analyzer on my string

analyzer :custom
tokenizer:whitespace
filter:stop
stop words:[is]

it broken down into a Tokenizer and remove stop words. if search for
'is' i found 0 hit because it remove from the string when creating an
index.
on other case i don't applied any analyzer mean it uses the default
analyzer on searching i have applied the same analyzer[custom] then
what will be the respected results.

please help me to clear things.if any stage i'm wrong then correct me.

Thanks


(system) #2