Hi,
On Tuesday, May 1, 2012 1:56:59 PM UTC-4, Sujoy Sett wrote:
We were trying jmeter testing on elasticsearch queries that are being used
in our application. We ran single user as well as five concurrent user
tests via jmeter.
Following are the findings:
- Regarding the data sample that I posted early in the mail trail, and
the kind of query I posted, a node of 2GB max heap size is being able to
serve a query n 100000 data volume. On increasing the data volume, the node
is facing OOM. *My question is, will dividing the data into more shards,
and adding more nodes (with same configuration), help me avoid hitting OOM?
Yes, adding more hardware will help. Which is not to say that you need
to add more hardware - it could be that you can avoid OOMs with your
existing hardware - maybe something can be tuned.
- I have used two configurations here, one - *multiple nodes in one
machine, with less heap space each node*. two - single node in one
machine, with more heap space. Which one is better in terms of
concurrent requests, heavy requests (terms facets), and what is the best
shard configuration?
I'd go with the latter.
- Regarding recovery from OOM, elasticsearch is showing random behavior.
We have switched off dumping heap to file. Still sometimes ES recovers from
OOM, sometimes not. How to ensure avoidance of OOM from requests only? I
mean something like when a query is causing a tending to OOM, identifying
and aborting that query only, without making ES unresponsive. Does it
sound absurd?
I think so, but maybe I'm missing something. I'm actually a bit suspicious
about the JVM recovering from an OOM. Can that really happen? I don't
think I've ever seen that happen - JVM OOMs when it really cannot perform
enough GC to make enough room for new objects. So I can't see how the JVM
could truly recover from OOM.
- Our ES installation has some 50 indexes in total. After a shutdown, it
typically takes some 5-10 minutes to get the green state, and before that,
queries tend to result in UnavailableShardException. Can we control or
speed up the recovery of some indexes on priority than others.
Yes. Have a look at this thread:
https://groups.google.com/forum/?fromgroups#!searchin/elasticsearch/es$20shard$20dance/elasticsearch/iyWxQaF-FGU/9nToZElZu2YJ
Otis
Performance Monitoring for Solr / Elasticsearch / HBase -
Thanks,
On Saturday, April 28, 2012 11:31:45 PM UTC+5:30, Sujoy Sett wrote:
Hi,
One quick observation, when a single node is maintained for a
cluster, recovery from OOM is happening normally, though it is not that
fast.
But when the cluster is having two nodes, upon OOM the nodes are coming
to a standstill (no response available, CPU usage minimal, memory blocked
to maximum allowed size). On shutting down one node, the other is returning
to responsive state.
We changed multi-cast discovery to uni-cast, played a little with
discovery timeout parameters, with no avail.
What are we missing here, any suggestions?Thanks and Regards,
On Friday, April 27, 2012 9:47:54 PM UTC+5:30, jagdeep singh wrote:
Hi Otis,
Thanks a lot for your response.
We will definitely try the approaches you have suggested and update
you soon.Thanks and Regards
JagdeepOn Apr 27, 9:12 pm, Otis Gospodnetic otis.gospodne...@gmail.com
wrote:Hi Sujoy,
Say hi to Ian from Otis please
And about monitoring - we've used SPM for Elasticsearch to see and
understand behaviour of ES cache(s). Since we can see trend graphs in
SPM
for ES, we can see how the cache size changes when we run queries vs.
when
we use sort vs. when we facet on field X or X and Y, etc. And we can
see
that on the per-node basis, too. So having and seeing this data over
time
also helps with your "Just out of inquisitiveness, what is ES doing
internally?" question.You can also clear FieldCache for a given field and set TTL on it.
And since you mention using this for tag cloud, normalizing your tags
to
reduce their cardinality will also help. We just did all this stuff
for a
large client (tag normalization, soft cache, cache clearing,
adjustment of
field types to those that use less memory, etc.) and SPM for ES came in
very handy, if I may say so!Otis
On Friday, April 27, 2012 6:42:35 AM UTC-4, Sujoy Sett wrote:
Hi,
Can u please explain how to check the field data cache ? Do I have
to set
anything to monitor explicitly?
I often use the mobz-elasticsearch-head-24935c4 plugin to monitor
cluster
state and health, I didn't find anything like
index.cache.field.max_size
there in the cluster_state details.Thanks and Regards,
On Friday, April 27, 2012 3:52:04 PM UTC+5:30, Rafał Kuć wrote:
Hello,
Did you look at the size of the field data cache after sending
the
example query ?Regards,
RafałW dniu piątek, 27 kwietnia 2012 12:15:38 UTC+2 użytkownik Sujoy Sett
napisał:Hi,
We have been using elasticsearch 0.19.2 for storing and analyzing
data
from social media blogs and forums. The data volume is going up to
500000 documents per index, and size of this volume of data in
Elasticsearch index is going up to 3 GB per index per node (all
shards). We always maintain the number of replicas 1 less than the
total number of nodes to ensure that a copy of all shards should
reside on every node at any instant. The number of shards are
generally 10 for the size of indexes we mentioned above.We try different queries on these data for advanced visualization
purpose, and mainly facets for showing trend charts or keyword
clouds.
Following are some example of the query we execute:
{
"query" : {
"match_all" : { }
},
"size" : 0,
"facets" : {
"tag" : {
"terms" : {
"field" : "nouns",
"size" : 100
},
"_cache":false
}
}
}{
"query" : {
"match_all" : { }
},
"size" : 0,
"facets" : {
"tag" : {
"terms" : {
"field" : "phrases",
"size" : 100
},
"_cache":false
}
}
}While executing such queries we often encounter heap space
shortage,
and the nodes becomes unresponsive. Our main concern is that the
nodes
do not recover to normal state even after dumping the heap to a
hprof
file. The node still consumes the maximum allocated memory as
shown in
task manager java.exe process, and the nodes remain unresponsive
until
we manually kill and restart them.ES Configuration 1:
Elasticsearch Version 0.19.2
2 Nodes, one on each physical server
Max heap size 6GB per node.
10 shards, 1 replica.ES Configuration 2:
Elasticsearch Version 0.19.2
6 Nodes, three on each physical server
Max heap size 2GB per node.
10 shards, 5 replica.Server Configuration:
Windows 7 64 bit
64 bit JVM
8 GB pysical memory
Dual Core processorFor both the configuration mentioned above Elasticsearch was
unable to
respond to the facet queries mentioned above, it was also unable to
recover when a query failed due to heap space shortage.We are facing this issue in our production environments, and
request
you to please suggest a better configuration or a different
approach
if required.The mapping of the data is we use is as follows:
(keyword1 is a customized keyword analyzer, similarly standard1 is
a
customized standard analyzer){
"properties": {
"adjectives": {
"type": "string",
"analyzer": "stop2"
},
"alertStatus": {
"type": "string",
"analyzer": "keyword1"
},
"assignedByUserId": {
"type": "integer",
"index": "analyzed"
},
"assignedByUserName": {
"type": "string",
"analyzer": "keyword1"
},
"assignedToDepartmentId": {
"type": "integer",
"index": "analyzed"
},
"assignedToDepartmentName": {
"type": "string",
"analyzer": "keyword1"
},
"assignedToUserId": {
"type": "integer",
"index": "analyzed"
},
"assignedToUserName": {
"type": "string",
"analyzer": "keyword1"
},
"authorJsonMetadata": {
"properties": {
"favourites": {
"type": "string"
},
"followers": {
"type": "string"
},
"following": {
"type": "string"
},
"likes": {
"type": "string"
},
"listed": {
"type": "string"
},
"subscribers": {
"type": "string"
},
"subscription": {
"type": "string"
},
"uploads": {
"type": "string"
},
"views": {
"type": "string"
}
}
},
"authorKloutDetails": {
"dynamic": "true",
"properties": {
"amplificationScore": {
"type": "string"
},
"authorKloutDetailsFound": {
"type": "string"
},
"description": {
"type": "string"
},
"influencees": {
"dynamic": "true",
"properties": {
"kscore": {
"type": "string"
},
"twitter_screen_name": {
"type": "string"
}
}
},
"influencers": {
"dynamic": "true",
"properties": {
"kscore": {
"type": "string"
},
"twitter_screen_name": {
"type": "string"
}
}
},
"kloutClass": {
"type": "string"
},
"kloutClassDescription": {
"type": "string"
},
"kloutScore": {
"type": "string"
},
"kloutScoreDescription": {
"type": "string"
},
"kloutTopic": {
"type": "string"
},
"slope": {
"type": "string"
},
"trueReach": {
"type": "string"
},
"twitterId": {
"type": "string"
},
"twitterScreenName": {
"type":...
read more »
