I'm trying to search across multiple indexes and I couldn't understand the
result of the TF/TDF function. I didn't expect for the indexes where the
term is more frequent to get penalized.
Here follows an example:
When searching for the term "alice" the document {"_index": "index2",
"_type": "type", "_id": "1"} got a score 0.8784157 while {"_index": "index1",
"_type": "type", "_id": "1"} got a score 0.4451987.
In my use case I got one index about sports and another about celebrities
and when I search for a celebrity documents across sports and celebrities
indexes, results from sports index tend to appear in first place due to the
explanation above (we have few celebrities documents in sports index). But
the point is that when searching for a celebrity I would expect results
from the celebrity index.
Is there any way to calculate the score not penalizing indexes where the
frequency of a term is higher?
I have never tried or looked at the code, but off the top of my head
perhaps the DFS query type would work:
Since the DFS query type calculates the TF/IDF values based on the values
in each individual shard, perhaps it ignores which index the shard belongs
to. Easy to test.
If not, the solution might be tricky. You can eliminate term length
normalization, but your issue is with the IDF. You can create your own
Similarity, but the best you can do is ignore the IDF, which probably would
not be ideal.
Ultimately, you can try script based scoring. The TF/IDF values are exposed
to the scripts, so you can try to apply some type of normalization
yourself. Kludgy and it would impact performance.
Hopefully DFS queries would work or someone else has a better idea!
Cheers,
Ivan
On Tue, Feb 25, 2014 at 12:00 PM, Luiz Guilherme Pais dos Santos < luizgpsantos@gmail.com> wrote:
Hi,
I'm trying to search across multiple indexes and I couldn't understand the
result of the TF/TDF function. I didn't expect for the indexes where the
term is more frequent to get penalized.
When searching for the term "alice" the document {"_index": "index2",
"_type": "type", "_id": "1"} got a score 0.8784157 while {"_index":
"index1", "_type": "type", "_id": "1"} got a score 0.4451987.
In my use case I got one index about sports and another about celebrities
and when I search for a celebrity documents across sports and celebrities
indexes, results from sports index tend to appear in first place due to the
explanation above (we have few celebrities documents in sports index). But
the point is that when searching for a celebrity I would expect results
from the celebrity index.
Is there any way to calculate the score not penalizing indexes where the
frequency of a term is higher?
Since the DFS query type calculates the TF/IDF values based on the values
in each individual shard, perhaps it ignores which index the shard belongs
to. Easy to test.
If not, the solution might be tricky. You can eliminate term length
normalization, but your issue is with the IDF. You can create your own
Similarity, but the best you can do is ignore the IDF, which probably would
not be ideal.
Ultimately, you can try script based scoring. The TF/IDF values are
exposed to the scripts, so you can try to apply some type of normalization
yourself. Kludgy and it would impact performance.
Hopefully DFS queries would work or someone else has a better idea!
Cheers,
Ivan
On Tue, Feb 25, 2014 at 12:00 PM, Luiz Guilherme Pais dos Santos < luizgpsantos@gmail.com> wrote:
Hi,
I'm trying to search across multiple indexes and I couldn't understand
the result of the TF/TDF function. I didn't expect for the indexes where
the term is more frequent to get penalized.
When searching for the term "alice" the document {"_index": "index2",
"_type": "type", "_id": "1"} got a score 0.8784157 while {"_index":
"index1", "_type": "type", "_id": "1"} got a score 0.4451987.
In my use case I got one index about sports and another about celebrities
and when I search for a celebrity documents across sports and celebrities
indexes, results from sports index tend to appear in first place due to the
explanation above (we have few celebrities documents in sports index). But
the point is that when searching for a celebrity I would expect results
from the celebrity index.
Is there any way to calculate the score not penalizing indexes where the
frequency of a term is higher?
Great, I am glad that it worked. I do not use multi-index searches, so I
was not sure if it would. Good to know that shards from different indices
can be aggregated with DFS queries.
--
Ivan
On Tue, Feb 25, 2014 at 6:04 PM, Luiz Guilherme Pais dos Santos < luizgpsantos@gmail.com> wrote:
Hi Ivan,
The DFS query then fetch worked very well!
Thank you!
Cheers,
Luiz Guilherme
On Tue, Feb 25, 2014 at 5:15 PM, Ivan Brusic ivan@brusic.com wrote:
Since the DFS query type calculates the TF/IDF values based on the values
in each individual shard, perhaps it ignores which index the shard belongs
to. Easy to test.
If not, the solution might be tricky. You can eliminate term length
normalization, but your issue is with the IDF. You can create your own
Similarity, but the best you can do is ignore the IDF, which probably would
not be ideal.
Ultimately, you can try script based scoring. The TF/IDF values are
exposed to the scripts, so you can try to apply some type of normalization
yourself. Kludgy and it would impact performance.
Hopefully DFS queries would work or someone else has a better idea!
Cheers,
Ivan
On Tue, Feb 25, 2014 at 12:00 PM, Luiz Guilherme Pais dos Santos < luizgpsantos@gmail.com> wrote:
Hi,
I'm trying to search across multiple indexes and I couldn't understand
the result of the TF/TDF function. I didn't expect for the indexes where
the term is more frequent to get penalized.
When searching for the term "alice" the document {"_index": "index2",
"_type": "type", "_id": "1"} got a score 0.8784157 while {"_index":
"index1", "_type": "type", "_id": "1"} got a score 0.4451987.
In my use case I got one index about sports and another about
celebrities and when I search for a celebrity documents across sports and
celebrities indexes, results from sports index tend to appear in first
place due to the explanation above (we have few celebrities documents in
sports index). But the point is that when searching for a celebrity I would
expect results from the celebrity index.
Is there any way to calculate the score not penalizing indexes where the
frequency of a term is higher?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.