To our surprise this had no effect on performance. Is this normal, or at
least expected for a field like "tags" which, as you'd suspect has a
variable number of tokens per field and a high number of distinct tokens
across the whole index.
It will not affect things if the problem you have is not the actual facet
calculation, but constant loading of the facet data. Are you indexing data
while running the facet requests? Is the node under load? Do you have any
special configuration to the field data?
To our surprise this had no effect on performance. Is this normal, or at
least expected for a field like "tags" which, as you'd suspect has a
variable number of tokens per field and a high number of distinct tokens
across the whole index.
On Friday, May 25, 2012 4:42:44 PM UTC-4, kimchy wrote:
It will not affect things if the problem you have is not the actual facet
calculation, but constant loading of the facet data.
By "constant loading of facet data" do you mean reading docs/field values
from new index segments that are being created due to new documents being
added?
Are you indexing data while running the facet requests? Is the node under
load? Do you have any special configuration to the field data?
Yes, indexing was happening while search performance tests were running.
These nodes have 16 CPU cores, the load was under 16 and CPUs were not 100%
utilized.
I did not test things with indexing turned off, but that would also not be
realistic because I need to see new docs pretty quickly after they are
indexed and indexing has to be constantly running
Any special configuration to the field data....? Hm, I don't think so.
To our surprise this had no effect on performance. Is this normal, or at
least expected for a field like "tags" which, as you'd suspect has a
variable number of tokens per field and a high number of distinct tokens
across the whole index.
On Friday, May 25, 2012 4:42:44 PM UTC-4, kimchy wrote:
It will not affect things if the problem you have is not the actual facet
calculation, but constant loading of the facet data.
By "constant loading of facet data" do you mean reading docs/field values
from new index segments that are being created due to new documents being
added?
Either that, or configuring the field cache with LRU based / soft based
configuration, which can cause data to be reloaded each time.
Are you indexing data while running the facet requests? Is the node under
load? Do you have any special configuration to the field data?
Yes, indexing was happening while search performance tests were running.
These nodes have 16 CPU cores, the load was under 16 and CPUs were not
100% utilized.
I did not test things with indexing turned off, but that would also not be
realistic because I need to see new docs pretty quickly after they are
indexed and indexing has to be constantly running
If new data is being indexed, and a refresh interval of 1 second, it means
that for new docs (segments) the field data needs to be loaded (and all
relevant search requests wait for it to be loaded). In 0.20 we will have a
warmer API that allows to pre-warm new segments, meaning that search
requests will not "suffer" loading the data.
Any special configuration to the field data....? Hm, I don't think so.
To our surprise this had no effect on performance. Is this normal, or
at least expected for a field like "tags" which, as you'd suspect has a
variable number of tokens per field and a high number of distinct tokens
across the whole index.
On Friday, May 25, 2012 4:42:44 PM UTC-4, kimchy wrote:
It will not affect things if the problem you have is not the actual
facet calculation, but constant loading of the facet data.
By "constant loading of facet data" do you mean reading docs/field values
from new index segments that are being created due to new documents being
added?
Either that, or configuring the field cache with LRU based / soft based
configuration, which can cause data to be reloaded each time.
Aha. I think we used soft during our tests.
By "reloaded each time" I assume you mean "reloaded from index each time
data is needed and is not in the cache".
That said, I don't think we saw any cache loading activity in SPM.... we'll
need to recheck to be sure, thanks for this pointer.
Are you indexing data while running the facet requests? Is the node under
load? Do you have any special configuration to the field data?
Yes, indexing was happening while search performance tests were running.
These nodes have 16 CPU cores, the load was under 16 and CPUs were not
100% utilized.
I did not test things with indexing turned off, but that would also not
be realistic because I need to see new docs pretty quickly after they are
indexed and indexing has to be constantly running
If new data is being indexed, and a refresh interval of 1 second, it means
that for new docs (segments) the field data needs to be loaded (and all
relevant search requests wait for it to be loaded). In 0.20 we will have a
warmer API that allows to pre-warm new segments, meaning that search
requests will not "suffer" loading the data.
So with regards to the original subject, your main point here is that
indexing while doing performance tests and having a low index refresh
interval are probably having such a dominant and negative effect on
performance that using Facet Filter limit doesn't actually show any
positive effects?
Thanks,
Otis
Any special configuration to the field data....? Hm, I don't think so.
To our surprise this had no effect on performance. Is this normal, or
at least expected for a field like "tags" which, as you'd suspect has a
variable number of tokens per field and a high number of distinct tokens
across the whole index.
On Friday, May 25, 2012 4:42:44 PM UTC-4, kimchy wrote:
It will not affect things if the problem you have is not the actual
facet calculation, but constant loading of the facet data.
By "constant loading of facet data" do you mean reading docs/field
values from new index segments that are being created due to new documents
being added?
Either that, or configuring the field cache with LRU based / soft based
configuration, which can cause data to be reloaded each time.
Aha. I think we used soft during our tests.
By "reloaded each time" I assume you mean "reloaded from index each time
data is needed and is not in the cache".
That said, I don't think we saw any cache loading activity in SPM....
we'll need to recheck to be sure, thanks for this pointer.
Are you indexing data while running the facet requests? Is the node under
load? Do you have any special configuration to the field data?
Yes, indexing was happening while search performance tests were running.
These nodes have 16 CPU cores, the load was under 16 and CPUs were not
100% utilized.
I did not test things with indexing turned off, but that would also not
be realistic because I need to see new docs pretty quickly after they are
indexed and indexing has to be constantly running
If new data is being indexed, and a refresh interval of 1 second, it
means that for new docs (segments) the field data needs to be loaded (and
all relevant search requests wait for it to be loaded). In 0.20 we will
have a warmer API that allows to pre-warm new segments, meaning that search
requests will not "suffer" loading the data.
So with regards to the original subject, your main point here is that
indexing while doing performance tests and having a low index refresh
interval are probably having such a dominant and negative effect on
performance that using Facet Filter limit doesn't actually show any
positive effects?
Yes, the time is spent on loading the data by the actual search requests.
Thanks,
Otis
Any special configuration to the field data....? Hm, I don't think so.
To our surprise this had no effect on performance. Is this normal, or
at least expected for a field like "tags" which, as you'd suspect has a
variable number of tokens per field and a high number of distinct tokens
across the whole index.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.