Hi, all we have switched from Sphinx to ElasticSearch, but we have big performance troubles. In Sphinx with the same amount of data we had response times between: 50ms - 70ms . Now our response times are very high: 300-500ms.
I am sending you the configs and settings of all we have. Can you please take a look and help us to become again response times under 70ms ?
If you want some other info, please write me:
------------ settings and OS env. --------------
ES Settings:
2 Nodes on 2 different hosts
Each index has 1 primary shard we have 2 shards each index
3,762 Total Shards
3,762 Successful Shards
85 Indices
20,347,989 Documents
40.5GB Size
<nabble_img src="ES.png" border="0"/>
elasticsearch.yml
index.cache.field.type: soft
indices.cache.filter.size: 50%
index.fielddata.cache: soft
index.cache.field.expire: 60m
indices.fielddata.cache.size: 50%
indices.fielddata.cache.expire : 60m
index.store.type: mmapfs
transport.tcp.compress: true;
bootstrap.mlockall: true
index.search.slowlog.threshold.query.warn: 10s
index.search.slowlog.threshold.query.info: 5s
index.search.slowlog.threshold.query.debug: 2s
index.search.slowlog.threshold.query.trace: 500ms
Example: We have an index for the Country DE and there are 1,5M documents. This index has 2 shards.
Starting of ES:
/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms32g -Xmx32g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.2.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/ -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch
OS:
24 Cores
80 GB of RAM
60 GB are used
Disk space: 1,2 TB
350 GB used / 780GB free
Disc type: SAS
Mysql is running also on this machine
Example Query: search for some City, where we give to ES the location_id:
{"query":{"multi_match":{"query":"faviczer":"indexAnalyzer","tie_breaker":0.3,"fuzziness":2,"operator":"or"}},"sort":{"ad_time_priority":{"order":"desc"},"price":{"order":"desc"},"order_priority":{"order":"desc"},"_score":{"order":"desc"}},"facets":{"company_id":{"facet_filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"terms":{"field":"company_facet","order":"count","script":"doc['company_facet'].value + '|' + doc['company_id'].value"}},"job_type_id":{"facet_filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"terms":{"field":"jobtype_facet","order":"term","all_terms":true}}},"filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"size":10,"from":0,"explain":false,"highlight":{"order":"score","require_field_match":false,"pre_tags":[""],"post_tags":["</b>"],"fields":{"description":{"type":"fvh","force_source":true,"no_match_size":200,"index_options":"offsets","fragment_size":200,"number_of_fragments":2,"matched_fields":["description","title"]}}}}
Response time on this query: > 400ms .. what is very very slow. We have disabled the facetes as well, but nothing has changed.
Thanks Nik