Elastic Search Query performance problem

Hi, all we have switched from Sphinx to ElasticSearch, but we have big performance troubles. In Sphinx with the same amount of data we had response times between: 50ms - 70ms . Now our response times are very high: 300-500ms.

I am sending you the configs and settings of all we have. Can you please take a look and help us to become again response times under 70ms ?

If you want some other info, please write me:

------------ settings and OS env. --------------

ES Settings:

2 Nodes on 2 different hosts

Each index has 1 primary shard we have 2 shards each index

3,762 Total Shards

3,762 Successful Shards

85 Indices

20,347,989 Documents

40.5GB Size

<nabble_img src="ES.png" border="0"/>

elasticsearch.yml

index.cache.field.type: soft

indices.cache.filter.size: 50%

index.fielddata.cache: soft

index.cache.field.expire: 60m

indices.fielddata.cache.size: 50%

indices.fielddata.cache.expire : 60m

index.store.type: mmapfs

transport.tcp.compress: true;

bootstrap.mlockall: true

index.search.slowlog.threshold.query.warn: 10s

index.search.slowlog.threshold.query.info: 5s

index.search.slowlog.threshold.query.debug: 2s

index.search.slowlog.threshold.query.trace: 500ms

Example: We have an index for the Country DE and there are 1,5M documents. This index has 2 shards.

Starting of ES:

/usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms32g -Xmx32g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError -Delasticsearch -Des.pidfile=/var/run/elasticsearch.pid -Des.path.home=/usr/share/elasticsearch -cp :/usr/share/elasticsearch/lib/elasticsearch-1.1.2.jar:/usr/share/elasticsearch/lib/:/usr/share/elasticsearch/lib/sigar/ -Des.default.config=/etc/elasticsearch/elasticsearch.yml -Des.default.path.home=/usr/share/elasticsearch -Des.default.path.logs=/var/log/elasticsearch -Des.default.path.data=/var/lib/elasticsearch -Des.default.path.work=/tmp/elasticsearch -Des.default.path.conf=/etc/elasticsearch org.elasticsearch.bootstrap.Elasticsearch

OS:

24 Cores

80 GB of RAM

60 GB are used

Disk space: 1,2 TB

350 GB used / 780GB free

Disc type: SAS

Mysql is running also on this machine

Example Query: search for some City, where we give to ES the location_id:

{"query":{"multi_match":{"query":"faviczer":"indexAnalyzer","tie_breaker":0.3,"fuzziness":2,"operator":"or"}},"sort":{"ad_time_priority":{"order":"desc"},"price":{"order":"desc"},"order_priority":{"order":"desc"},"_score":{"order":"desc"}},"facets":{"company_id":{"facet_filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"terms":{"field":"company_facet","order":"count","script":"doc['company_facet'].value + '|' + doc['company_id'].value"}},"job_type_id":{"facet_filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"terms":{"field":"jobtype_facet","order":"term","all_terms":true}}},"filter":{"bool":{"must":[{"terms":{"sponsored":[1,0]}},{"not":{"filter":{"terms":{"job_source_id":[188]}}}}]}},"size":10,"from":0,"explain":false,"highlight":{"order":"score","require_field_match":false,"pre_tags":[""],"post_tags":["</b>"],"fields":{"description":{"type":"fvh","force_source":true,"no_match_size":200,"index_options":"offsets","fragment_size":200,"number_of_fragments":2,"matched_fields":["description","title"]}}}}

Response time on this query: > 400ms .. what is very very slow. We have disabled the facetes as well, but nothing has changed.

Thanks Nik