Slow ElasticSearch queries and huge load

Elastic Stack Elasticsearch
1

Hello,

I saw already some topics about the performance of ES here, but none of
them answered any of my questions. I would like to use ES for a search
engine. It looks really great how easy it is to create new nodes, connect
them together into clusters, make replicas etc. comparing to Solr or
Sphinx. I've created 3 nodes with 3 shards and 1 replica (+1 ElasticSearch
for balancing only). Each node is run with such JVM params:

-Xms5g -Xmx5g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

Machine has 16GB or RAM, 4 cores Intel i5 processor @ 3.40 GHz

There is only one index with one mapping which look like that:

{
"files" : {
"groups" : {
"_source" : {
"enabled" : false
},
"properties" : {
"id" : {
"type" : "string",
"index" : "no",
"store" : "yes"
},
"clicks" : {
"type" : "integer"
},
"date_added" : {
"type" : "long"
},
"desc" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"extension_attr" : {
"type" : "short"
},
"group_id" : {
"type" : "long",
"store" : "yes"
},
"host" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"hosting_id" : {
"type" : "integer"
},
"name" : {
"type" : "string",
"boost" : 3.0,
"term_vector" : "with_positions_offsets"
},
"size" : {
"type" : "integer"
},
"source_title" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"source_url" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
}
}
}
}
}

In this index there are around 30M records for both ElasticSearch and
Sphinx Search. Whether Sphinx on the same machine and single node makes
response for a MATCH query like MATCH('test') takes around 0.14s then in
ElasticSearch it's 2,3s. I'm surprised of these results. I'm not sure if
it's normal time for a query which does not reside in cache. I'm thinking
if there's any possibility to tune ES.

--

2

Forgot to put the exact query:

{
"query": {
"custom_score": {
"query": {
"query_string": {
"default_field": "_all",
"default_operator": "and",
"query": "test"
}
}
}
},
"fields": [
"group_id",
"name",
"date_added"
],
"from": 0,
"size": 50,
"sort": ,
"facets": {}
}

Is this normal to have responses around 2s for such a query?

W dniu piątek, 7 grudnia 2012 15:45:54 UTC+1 użytkownik Piotr napisał:

Hello,

I saw already some topics about the performance of ES here, but none of
them answered any of my questions. I would like to use ES for a search
engine. It looks really great how easy it is to create new nodes, connect
them together into clusters, make replicas etc. comparing to Solr or
Sphinx. I've created 3 nodes with 3 shards and 1 replica (+1 Elasticsearch
for balancing only). Each node is run with such JVM params:

-Xms5g -Xmx5g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

Machine has 16GB or RAM, 4 cores Intel i5 processor @ 3.40 GHz

There is only one index with one mapping which look like that:

{
"files" : {
"groups" : {
"_source" : {
"enabled" : false
},
"properties" : {
"id" : {
"type" : "string",
"index" : "no",
"store" : "yes"
},
"clicks" : {
"type" : "integer"
},
"date_added" : {
"type" : "long"
},
"desc" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"extension_attr" : {
"type" : "short"
},
"group_id" : {
"type" : "long",
"store" : "yes"
},
"host" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"hosting_id" : {
"type" : "integer"
},
"name" : {
"type" : "string",
"boost" : 3.0,
"term_vector" : "with_positions_offsets"
},
"size" : {
"type" : "integer"
},
"source_title" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"source_url" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
}
}
}
}
}

In this index there are around 30M records for both Elasticsearch and
Sphinx Search. Whether Sphinx on the same machine and single node makes
response for a MATCH query like MATCH('test') takes around 0.14s then in
Elasticsearch it's 2,3s. I'm surprised of these results. I'm not sure if
it's normal time for a query which does not reside in cache. I'm thinking
if there's any possibility to tune ES.

--

3

There could be many reasons for this and one would have to dig in. The
most basic thing that comes to mind is that on this 1 server you likely
have 5 index shards (default), which is pointless and adds overhead. Make
it 1, reindex, and then try. I'd love to see the before and after
comparison.

Otis

ELASTICSEARCH Performance Monitoring - Sematext Monitoring | Infrastructure Monitoring Service
Search Analytics - Cloud Monitoring Tools & Services | Sematext

On Friday, December 7, 2012 9:45:54 AM UTC-5, Piotr wrote:

Hello,

I saw already some topics about the performance of ES here, but none of
them answered any of my questions. I would like to use ES for a search
engine. It looks really great how easy it is to create new nodes, connect
them together into clusters, make replicas etc. comparing to Solr or
Sphinx. I've created 3 nodes with 3 shards and 1 replica (+1 Elasticsearch
for balancing only). Each node is run with such JVM params:

-Xms5g -Xmx5g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC
-XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly -XX:+HeapDumpOnOutOfMemoryError
-XX:+UseTLAB -XX:+CMSClassUnloadingEnabled -XX:+CMSPermGenSweepingEnabled

Machine has 16GB or RAM, 4 cores Intel i5 processor @ 3.40 GHz

There is only one index with one mapping which look like that:

{
"files" : {
"groups" : {
"_source" : {
"enabled" : false
},
"properties" : {
"id" : {
"type" : "string",
"index" : "no",
"store" : "yes"
},
"clicks" : {
"type" : "integer"
},
"date_added" : {
"type" : "long"
},
"desc" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"extension_attr" : {
"type" : "short"
},
"group_id" : {
"type" : "long",
"store" : "yes"
},
"host" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"hosting_id" : {
"type" : "integer"
},
"name" : {
"type" : "string",
"boost" : 3.0,
"term_vector" : "with_positions_offsets"
},
"size" : {
"type" : "integer"
},
"source_title" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
},
"source_url" : {
"type" : "string",
"term_vector" : "with_positions_offsets"
}
}
}
}
}

In this index there are around 30M records for both Elasticsearch and
Sphinx Search. Whether Sphinx on the same machine and single node makes
response for a MATCH query like MATCH('test') takes around 0.14s then in
Elasticsearch it's 2,3s. I'm surprised of these results. I'm not sure if
it's normal time for a query which does not reside in cache. I'm thinking
if there's any possibility to tune ES.

--

4

In this index there are around 30M records for both Elasticsearch and
Sphinx Search.
Whether Sphinx on the same machine and single node makes response for a
MATCH query like MATCH('test') takes around 0.14s then in Elasticsearch
it's 2,3s.
I'm surprised of these results. I'm not sure if it's normal time for a
query which does not reside in cache.

First, it's important to understand if you're interested in numbers for a
"cold" system, or a "warmed up" system, where Elasticsearch and OS caches
are filled and used. Usually, the "cold" numbers are not of much use as a
guideline for production use; note, that recent Elasticsearch versions come
with a Warmer API 1.

Second, just send the same request repeatedly, and observe if the took
time and the response time is decreasing and how much. Observe
Elasticsearch with a tool such as Paramedic or BigDesk to visually check
the amount of memory needed, CPU, etc.

Third, as Otis hints, Elasticsearch by default splits your index into 5
shards (in effect, separate Lucene indices). It may be a contributing
factor, try to create the index with one shard only and no replicas:

curl -X PUT localhost:9200/my-index/_settings -d '{ "index" :

{"number_of_shards": 1, "number_of_replicas": 0} }'

Last, Elasticsearch can log slow queries 2, when you start testing more
complex queries, it may be useful for you to see the took time for each
query.

Karel

--