Elastic Search Query performance problem


(nnikolay) #1

Hi, all we have switched from Sphinx to ElasticSearch, but we have big performance troubles. In Sphinx with the same amount of data we had response times between: 50ms - 70ms . Now our response times are very high: 300-500ms.

I am sending you the configs and settings of all we have. Can you please take a look and help us to become again response times under 70ms ?

If you want some other info, please write me:

------------ settings and OS env. --------------

ES Settings:

2 Nodes on 2 different hosts

Each index has 1 primary shard we have 2 shards each index

3,762 Total Shards

3,762 Successful Shards

85 Indices

20,347,989 Documents

40.5GB Size

<nabble_img src="ES.png" border="0"/>

elasticsearch.yml

index.cache.field.type: soft

indices.cache.filter.size: 50%

index.fielddata.cache: soft

index.cache.field.expire: 60m

indices.fielddata.cache.size: 50%

indices.fielddata.cache.expire : 60m

index.store.type: mmapfs

transport.tcp.compress: true;

bootstrap.mlockall: true

index.search.slowlog.threshold.query.warn: 10s

index.search.slowlog.threshold.query.info: 5s

index.search.slowlog.threshold.query.debug: 2s

index.search.slowlog.threshold.query.trace: 500ms

Example: We have an index for the Country DE and there are 1,5M documents. This index has 2 shards.

Starting of ES:

/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms32g -Xmx32g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.2.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/ -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch

OS:

24 Cores

80 GB of RAM

60 GB are used

Disk space: 1,2 TB

350 GB used / 780GB free

Disc type: SAS

Mysql is running also on this machine

Example Query: search for some City, where we give to ES the location_id:

{"query":{"multi_match":{"query":"faviczer":"indexAnalyzer","tie_breaker":0.3,"fuzziness":2,"operator":"or"}},"sort":{"ad_time_priority":{"order":"desc"},"price":{"order":"desc"},"order_priority":{"order":"desc"},"_score":{"order":"desc"}},"facets":{"company_id":{"facet_filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"terms":{"field":"company_facet","order":"count","script":"doc['company_facet'].value + '|' + doc['company_id'].value"}},"job_type_id":{"facet_filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"terms":{"field":"jobtype_facet","order":"term","all_terms":true}}},"filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"size":10,"from":0,"explain":false,"highlight":{"order":"score","require_field_match":false,"pre_tags":[""],"post_tags":["</b>"],"fields":{"description":{"type":"fvh","force_source":true,"no_match_size":200,"index_options":"offsets","fragment_size":200,"number_of_fragments":2,"matched_fields":["description","title"]}}}}

Response time on this query: > 400ms .. what is very very slow. We have disabled the facetes as well, but nothing has changed.

Thanks Nik


(system) #2