Hello,
We run an elastic search cluster that we've been experiencing random slow
queries on. Any help in debugging the source of the slow query would be
appreciated. Basically, we are getting occasional searches that are 1.5 ->
12 seconds. Most searches are much faster. The goal would be to return
search results in less than 1 second.
We run Elastic Search 0.90.2 that we've recently installed, and we
re-indexed all of our data (as opposed to upgrading the existing .19.11
systems). We have over 180,000,000 documents. We are configured to run 5
shards with 1 replica. Our systems are deployed in Amazon EC2 on 2
m1.xlarge instances. Both Elastic Search cluster members run with a heap
of 10Gig. CPU and I/O load on the systems are very low. Indexing
performance seems excellent. We run a refresh interval of 10 seconds.
Current disk usage for the data is under 140gig across both search
instances.
Here is our schema:
{
"cluster_name": "Open",
"master_node": "2MascUtjSCmSLyrGlxiBYg",
"blocks": {},
"nodes": {
"2MascUtjSCmSLyrGlxiBYg": {
"name": "Black Panther",
"transport_address": "inet[/10.0.4.51:9300]",
"attributes": {}
},
"HTGsGWe9QfaqdJbvzE9qaw": {
"name": "Darkdevil",
"transport_address": "inet[/10.0.8.51:9300]",
"attributes": {}
}
},
"metadata": {
"templates": {},
"indices": {
"people_v1": {
"state": "open",
"settings": {
"index.number_of_shards": "5",
"index.number_of_replicas": "1",
"index.version.created": "900299",
"index.analysis.filter.my_ngram.min_gram": "1",
"index.analysis.filter.my_metaphone.replace": "true",
"index.analysis.filter.my_metaphone.type": "phonetic",
"index.analysis.analyzer.metaphone_analyzer.tokenizer":
"standard",
"index.analysis.filter.nickname_filter.synonyms_path":
"analysis/names1.2.csv",
"index.analysis.filter.nickname_filter.type": "synonym",
"index.analysis.analyzer.ngram_analyzer.tokenizer":
"standard",
"index.analysis.filter.my_ngram.type": "edgeNGram",
"index.analysis.analyzer.email_analyzer.tokenizer":
"uax_url_email",
"index.analysis.analyzer.nickname_analyzer.tokenizer":
"standard",
"index.analysis.filter.my_metaphone.encoder":
"metaphone",
"index.analysis.analyzer.standard_analyzer.tokenizer":
"standard",
"index.analysis.analyzer.metaphone_analyzer.filter.1":
"lowercase",
"index.analysis.analyzer.metaphone_analyzer.filter.0":
"standard",
"index.analysis.filter.my_ngram.max_gram": "5",
"index.analysis.analyzer.nickname_analyzer.filter.0":
"standard",
"index.analysis.analyzer.domain_analyzer.tokenizer":
"uax_url_email",
"index.analysis.analyzer.ngram_analyzer.filter.1":
"lowercase",
"index.analysis.analyzer.ngram_analyzer.filter.0":
"standard",
"index.analysis.analyzer.nickname_analyzer.filter.2":
"nickname_filter",
"index.analysis.analyzer.nickname_analyzer.filter.1":
"lowercase",
"index.analysis.analyzer.standard_analyzer.filter.1":
"lowercase",
"index.analysis.analyzer.metaphone_analyzer.filter.2":
"my_metaphone",
"index.analysis.analyzer.standard_analyzer.filter.0":
"standard",
"index.analysis.analyzer.ngram_analyzer.filter.2":
"my_ngram",
"index.refresh_interval": "10s"
},
"mappings": {
"person": {
"_source": {
"compress": true
},
"_routing": {
"path": "userid",
"required": true
},
"properties": {
"email": {
"analyzer": "email_analyzer",
"type": "string"
},
"nameOf": {
"index": "no",
"type": "string"
},
"company": {
"type": "multi_field",
"fields": {
"company_ngram": {
"include_in_all": false,
"analyzer": "ngram_analyzer",
"type": "string"
},
"company": {
"analyzer": "standard",
"type": "string"
}
}
},
"userid": {
"index": "not_analyzed",
"omit_norms": true,
"index_options": "docs",
"type": "string"
},
"score": {
"type": "integer"
},
"domain": {
"analyzer": "domain_analyzer",
"type": "string"
},
"ownerid": {
"index": "no",
"type": "string"
},
"ownernameof": {
"index": "no",
"type": "string"
},
"key": {
"index": "not_analyzed",
"omit_norms": true,
"index_options": "docs",
"type": "string"
},
"namepart": {
"type": "multi_field",
"fields": {
"nickname": {
"include_in_all": false,
"analyzer": "nickname_analyzer",
"type": "string"
},
"metaphone": {
"include_in_all": false,
"analyzer": "metaphone_analyzer",
"boost": 0.8,
"type": "string"
},
"ngram": {
"include_in_all": false,
"analyzer": "ngram_analyzer",
"type": "string"
},
"namepart": {
"analyzer": "standard_analyzer",
"type": "string"
}
}
}
}
}
},
"aliases": []
}
}
},
We use routing in our implementation, and all the data being searched are
segmented by userid.
Our current searches are similar to the following:
curl -XPOST
'http://localhost:9200/people_v1/person/_search?pretty=true&routing=9b3ac7e80c7968b8ca1d6e56f96a86c6'
-d '
{
size: 9,
sort:
{
score:
{
order: "desc"
}
},
query:
{
filtered:
{
query:
{
query_string: { query:
"userid:9b3ac7e80c7968b8ca1d6e56f96a86c6" }
},
filter:
{
and:[
{
or:[
{
term:
{
ngram: "as"
}
},
{
term:
{
company_ngram: "as"
}
},
{
query:
{
match:
{
nickname: "m"
}
}
}]
},
{
not:
{
term:
{
key:"9b3ac7e80c7968b8ca1d6e56f96a86c605d73158bcf9968d0d6bcd99db050f4b"
}
}
}]
}
}
}
}
}'
I've tried different mechanisms of searching, but still experience the
random slowness. For instance, this query exhibits the same behavior:
curl -XPOST
'http://localhost:9200/people_v1/person/_search?pretty=true&routing=9b3ac7e80c7968b8ca1d6e56f96a86c6'
-d '
{
size: 9,
query:
{
filtered:
{
query:
{
custom_score:
{
query:
{
term: { userid: "9b3ac7e80c7968b8ca1d6e56f96a86c6" }
},
script: "_source.score"
}
},
filter:
{
and:[
{
or:[
{
prefix:
{
namepart: "chuck"
}
},
{
prefix:
{
company: "chuck"
}
},
{
query:
{
match:
{
nickname: "chuck"
}
}
}]
},
{
not:
{
term:
{
key:"9b3ac7e80c7968b8ca1d6e56f96a86c605d73158bcf9968d0d6bcd99db050f4b"
}
}
}]
}
}
}
}
}'
Our results need to be sorted by a "score" that we pre-compute (the query
above attempts to use that score). Our searches use an edge ngram to
retrieve results up until 5 characters, and then we start using prefix
queries. Essentially searches are getting results based on name, nickname,
and company name, sorted by score.
Our results from "curl -XGET 'http://localhost:9200/_segments?pretty=true'"
show 17->30 search segments per shard. I've read some advice online that
running optimize may help, although I've read other areas that say not to
run optimize, it will get you into trouble, and let Lucene handle it.
Any advice on how we can debug this issue/improve performance?
Thanks,
-Mike
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.
