Query timings breakdown and performance issues

Elastic Stack Elasticsearch
1

Having a few issues with search performance. I was hoping there would be
some way to get a detailed output on what phases of the search took all my
time. I'm planning on passing the slowlog along to Kibana to set some idea
of what is going on on a shard level, but thought I'd put this out there
incase there was any way to get a breakdown along with the query response.
i.e. I had to wait on shard X due to Y. Plus for highlighting any flaws
in what I'm doing.

Setup: 6 data nodes, 256GB, 8xSSD, 60GB heap, G1GC

Main search index: ~8TB + 1 replica. ~600mil docs (news articles)

Other indices: 1TB + replicas, ~1bil docs

Indexing: 150 docs/sec

Querying: Full text, date histogram, faceting.

Problems:

  • Some of our queries are quite large. Large near queries, up to 100
    terms or'd together, neared with 5 terms, including wildcards. That type
    of thing. They take a long time, sometimes upwards of 3 mins. We want to
    be able to support this. 1 second responses maybe not, but I'd say 30
    seconds at a max..
  • These slow queries tend to prevent other queries from completing in
    sensible times, even for simple queries.
  • There's a large variance in query times. Sometimes 1 sec, sometimes
    much higher.

I've tried to correlate to GCs and merges without success. Ideally I'd
like to know what were queries waiting on, from which I can make
adjustments. Obviously I could add more nodes to the cluster, but I'd like
to know why I'm doing that rather than just taking a punt. Advise/abuse
welcome.

Harry

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1a668bd4-86a8-42d4-820c-a29d1be68762%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

I'd be curious to see an example of such a query! :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1d1862c7-b0cf-4826-8cda-d02e55cd552b%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

No you don't Binh

On Wednesday, 9 April 2014 17:57:17 UTC+1, Binh Ly wrote:

I'd be curious to see an example of such a query! :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/33a07c88-54f4-4363-917f-bf3e75d6cb5e%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

I know, it's hideous.

They typically consist of an "entity" neared with a large number of
"negative" (user specified) terms. So something like [bad person near50
(long or list near5 negative or terms ...)] which we translate into a big
horrible near_span query. When we can detect a language then we can apply
one analyzer and query the appropriate fields. Otherwise, we query across
a number of fields with different language analyzed fields. It's not nice.

It's pretty typical in certain industries.

On Thursday, 10 April 2014 00:03:48 UTC+1, hazzadous wrote:

No you don't Binh

On Wednesday, 9 April 2014 17:57:17 UTC+1, Binh Ly wrote:

I'd be curious to see an example of such a query! :slight_smile:

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b4344c78-160c-4b75-9945-e7719f387735%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.