if it can help :
_nodes/hot_threads?pretty=1
::: [Sepulchre][tcr0m37nRAS9appE1pvWLA][sd-70975][inet[/x.x.x.x:9300]]
20.6% (102.7ms out of 500ms) cpu usage by thread
'elasticsearch[Sepulchre][search][T#78]'
8/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.LinkedTransferQueue.awaitMatch(
LinkedTransferQueue.java:735)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.
java:644)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.
java:1137)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(
SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
2/10 snapshots sharing following 15 elements
org.apache.lucene.search.TopFieldCollector.populateResults(
TopFieldCollector.java:1203)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:156)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:93)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:582)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:275)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
20.2% (101ms out of 500ms) cpu usage by thread
'elasticsearch[Sepulchre][search][T#4]'
10/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.LinkedTransferQueue.awaitMatch(
LinkedTransferQueue.java:735)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.
java:644)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.
java:1137)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(
SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
19.3% (96.4ms out of 500ms) cpu usage by thread
'elasticsearch[Sepulchre][search][T#42]'
9/10 snapshots sharing following 10 elements
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
java.util.concurrent.LinkedTransferQueue.awaitMatch(
LinkedTransferQueue.java:735)
java.util.concurrent.LinkedTransferQueue.xfer(LinkedTransferQueue.
java:644)
java.util.concurrent.LinkedTransferQueue.take(LinkedTransferQueue.
java:1137)
org.elasticsearch.common.util.concurrent.SizeBlockingQueue.take(
SizeBlockingQueue.java:162)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.
java:1068)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1130)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
unique snapshot
org.apache.lucene.search.
FieldValueHitQueue$MultiComparatorsFieldValueHitQueue.lessThan(
FieldValueHitQueue.java:122)
org.apache.lucene.search.
FieldValueHitQueue$MultiComparatorsFieldValueHitQueue.lessThan(
FieldValueHitQueue.java:99)
org.apache.lucene.util.PriorityQueue.downHeap(PriorityQueue.java:241)
org.apache.lucene.util.PriorityQueue.pop(PriorityQueue.java:177)
org.apache.lucene.search.TopFieldCollector.populateResults(
TopFieldCollector.java:1203)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:156)
org.apache.lucene.search.TopDocsCollector.topDocs(TopDocsCollector.
java:93)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:582)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:533)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:510)
org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:345)
org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:150
)
org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:275)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:231)
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(
SearchServiceTransportAction.java:228)
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(
SearchServiceTransportAction.java:559)
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.
java:1145)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor
.java:615)
java.lang.Thread.run(Thread.java:745)
_nodes/stats?pretty=1
{
"cluster_name" : "elasticsearch",
"nodes" : {
"tcr0m37nRAS9appE1pvWLA" : {
"timestamp" : 1422495166302,
"name" : "Sepulchre",
"transport_address" : "inet[/x.x.x.x:9300]",
"host" : "sd-70975",
"ip" : [ "inet[/x.x.x.x:9300]", "NONE" ],
"indices" : {
"docs" : {
"count" : 936269,
"deleted" : 545
},
"store" : {
"size_in_bytes" : 856723820,
"throttle_time_in_millis" : 36
},
"indexing" : {
"index_total" : 49,
"index_time_in_millis" : 1058,
"index_current" : 0,
"delete_total" : 184,
"delete_time_in_millis" : 84,
"delete_current" : 0,
"noop_update_total" : 0,
"is_throttled" : false,
"throttle_time_in_millis" : 0
},
"get" : {
"total" : 0,
"time_in_millis" : 0,
"exists_total" : 0,
"exists_time_in_millis" : 0,
"missing_total" : 0,
"missing_time_in_millis" : 0,
"current" : 0
},
"search" : {
"open_contexts" : 16,
"query_total" : 11557772,
"query_time_in_millis" : 91259801,
"query_current" : 9,
"fetch_total" : 10139269,
"fetch_time_in_millis" : 37413865,
"fetch_current" : 0
},
"merges" : {
"current" : 0,
"current_docs" : 0,
"current_size_in_bytes" : 0,
"total" : 5,
"total_time_in_millis" : 5835,
"total_docs" : 7258,
"total_size_in_bytes" : 12852963
},
"refresh" : {
"total" : 287,
"total_time_in_millis" : 5430
},
"flush" : {
"total" : 45,
"total_time_in_millis" : 1016
},
"warmer" : {
"current" : 0,
"total" : 445,
"total_time_in_millis" : 308
},
"filter_cache" : {
"memory_size_in_bytes" : 204470568,
"evictions" : 0
},
"id_cache" : {
"memory_size_in_bytes" : 0
},
"fielddata" : {
"memory_size_in_bytes" : 6131132,
"evictions" : 0
},
"percolate" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0,
"memory_size_in_bytes" : -1,
"memory_size" : "-1b",
"queries" : 0
},
"completion" : {
"size_in_bytes" : 0
},
"segments" : {
"count" : 679,
"memory_in_bytes" : 41166270,
"index_writer_memory_in_bytes" : 0,
"index_writer_max_memory_in_bytes" : 3168274004,
"version_map_memory_in_bytes" : 3268,
"fixed_bit_set_memory_in_bytes" : 0
},
"translog" : {
"operations" : 122,
"size_in_bytes" : 17
},
"suggest" : {
"total" : 0,
"time_in_millis" : 0,
"current" : 0
},
"query_cache" : {
"memory_size_in_bytes" : 0,
"evictions" : 0,
"hit_count" : 0,
"miss_count" : 0
}
},
"os" : {
"timestamp" : 1422495166542,
"uptime_in_millis" : 10821,
"load_average" : [ 49.66, 47.58, 46.68 ],
"cpu" : {
"sys" : 3,
"user" : 83,
"idle" : 12,
"usage" : 86,
"stolen" : 0
},
"mem" : {
"free_in_bytes" : 221818286080,
"used_in_bytes" : 48629456896,
"free_percent" : 84,
"used_percent" : 15,
"actual_free_in_bytes" : 228507299840,
"actual_used_in_bytes" : 41940443136
},
"swap" : {
"used_in_bytes" : 0,
"free_in_bytes" : 10238291968
}
},
"process" : {
"timestamp" : 1422495166543,
"open_file_descriptors" : 2162,
"cpu" : {
"percent" : 982,
"sys_in_millis" : 3025760,
"user_in_millis" : 103162570,
"total_in_millis" : 106188330
},
"mem" : {
"resident_in_bytes" : 7449333760,
"share_in_bytes" : 75313152,
"total_virtual_in_bytes" : 53864665088
}
},
"jvm" : {
"timestamp" : 1422495166630,
"uptime_in_millis" : 10807489,
"mem" : {
"heap_used_in_bytes" : 4758146816,
"heap_used_percent" : 15,
"heap_committed_in_bytes" : 30894260224,
"heap_max_in_bytes" : 30894260224,
"non_heap_used_in_bytes" : 54495664,
"non_heap_committed_in_bytes" : 54984704,
"pools" : {
"young" : {
"used_in_bytes" : 536440,
"max_in_bytes" : 1954217984,
"peak_used_in_bytes" : 1954217984,
"peak_max_in_bytes" : 1954217984
},
"survivor" : {
"used_in_bytes" : 28468776,
"max_in_bytes" : 244252672,
"peak_used_in_bytes" : 151595664,
"peak_max_in_bytes" : 244252672
},
"old" : {
"used_in_bytes" : 4729141600,
"max_in_bytes" : 28695789568,
"peak_used_in_bytes" : 4729141600,
"peak_max_in_bytes" : 28695789568
}
}
},
"threads" : {
"count" : 346,
"peak_count" : 362
},
"gc" : {
"collectors" : {
"young" : {
"collection_count" : 16969,
"collection_time_in_millis" : 1384852
},
"old" : {
"collection_count" : 0,
"collection_time_in_millis" : 0
}
}
},
"buffer_pools" : {
"direct" : {
"count" : 1673,
"used_in_bytes" : 70117388,
"total_capacity_in_bytes" : 70117388
},
"mapped" : {
"count" : 314,
"used_in_bytes" : 154987920,
"total_capacity_in_bytes" : 154987920
}
}
},
"thread_pool" : {
"generic" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 4,
"completed" : 1443
},
"index" : {
"threads" : 32,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 32,
"completed" : 233
},
"bench" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"get" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"snapshot" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"merge" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 5,
"completed" : 116
},
"suggest" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"bulk" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"optimize" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"warmer" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 2,
"completed" : 208
},
"flush" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 5,
"completed" : 113
},
"search" : {
"threads" : 96,
"queue" : 1,
"active" : 9,
"rejected" : 0,
"largest" : 96,
"completed" : 24008797
},
"listener" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"percolate" : {
"threads" : 0,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 0,
"completed" : 0
},
"management" : {
"threads" : 5,
"queue" : 0,
"active" : 1,
"rejected" : 0,
"largest" : 5,
"completed" : 63903
},
"refresh" : {
"threads" : 1,
"queue" : 0,
"active" : 0,
"rejected" : 0,
"largest" : 6,
"completed" : 112
}
},
"network" : {
"tcp" : {
"active_opens" : 2312428,
"passive_opens" : 2484460,
"curr_estab" : 582,
"in_segs" : 209057765,
"out_segs" : 216923023,
"retrans_segs" : 49717,
"estab_resets" : 1144,
"attempt_fails" : 135,
"in_errs" : 34,
"out_rsts" : 2459
}
},
"fs" : {
"timestamp" : 1422495166632,
"total" : {
"total_in_bytes" : 302674501632,
"free_in_bytes" : 264467443712,
"available_in_bytes" : 249068793856,
"disk_reads" : 351417,
"disk_writes" : 205904,
"disk_io_op" : 557321,
"disk_read_size_in_bytes" : 3433067520,
"disk_write_size_in_bytes" : 3127025664,
"disk_io_size_in_bytes" : 6560093184,
"disk_queue" : "0",
"disk_service_time" : "0.1"
},
"data" : [ {
"path" : "/var/lib/elasticsearch/elasticsearch/nodes/0",
"mount" : "/",
"dev" : "/dev/sda2",
"total_in_bytes" : 302674501632,
"free_in_bytes" : 264467443712,
"available_in_bytes" : 249068793856,
"disk_reads" : 351417,
"disk_writes" : 205904,
"disk_io_op" : 557321,
"disk_read_size_in_bytes" : 3433067520,
"disk_write_size_in_bytes" : 3127025664,
"disk_io_size_in_bytes" : 6560093184,
"disk_queue" : "0",
"disk_service_time" : "0.1"
} ]
},
"transport" : {
"server_open" : 13,
"rx_count" : 6,
"rx_size_in_bytes" : 1380,
"tx_count" : 6,
"tx_size_in_bytes" : 1380
},
"http" : {
"current_open" : 11,
"total_opened" : 2311818
},
"breakers" : {
"request" : {
"limit_size_in_bytes" : 12357704089,
"limit_size" : "11.5gb",
"estimated_size_in_bytes" : 16440,
"estimated_size" : "16kb",
"overhead" : 1.0,
"tripped" : 0
},
"fielddata" : {
"limit_size_in_bytes" : 18536556134,
"limit_size" : "17.2gb",
"estimated_size_in_bytes" : 6131132,
"estimated_size" : "5.8mb",
"overhead" : 1.03,
"tripped" : 0
},
"parent" : {
"limit_size_in_bytes" : 21625982156,
"limit_size" : "20.1gb",
"estimated_size_in_bytes" : 6147572,
"estimated_size" : "5.8mb",
"overhead" : 1.0,
"tripped" : 0
}
}
}
}
}
On Wednesday, January 28, 2015 at 11:41:16 PM UTC+1, Oto Iashvili wrote:
Hi,
thanks a lot for answer,
Ive tried several value for heap, between 26 and 32, but I didnt see any
difference.
I remove G1 and put back default parameter. But still same pb.
I was saying around 500%, but it is just average, i goes sometimes up to
2000%
I was also thinking to take several server, but right now it is not
possible.
I was using a smaller server just before, with 96go of ram and I was
working better. I trie to put same parameters as before, but it is not much
slower.
On Wednesday, January 28, 2015 at 2:36:29 PM UTC+1, Jilles van Gurp wrote:
How much heap are you giving to ES? With this many requests, if your
setup is not falling over it is probably not garbage collect related
because that would result in very noticable delays/unavailability of es.
32GB should be a good value given how much memory you have. Also, you
probably want to use doc_values in your mapping so that you can utilize the
os file cache and move some of the memory pressure on the heap. You seem to
have plenty of ram, so your entire dataset should easily fit in RAM.
Also, don't use G1 for elasticsearch. There are known issues with that
particular garbage collector in combination with lucene. CMS is the best
option for ES.
500% of 20 cores doesn't sound that bad; you'd max them out at 4000%.
Still, it would be nice to know what it is doing. In any case, you might
want to try out marvel to find out where your setup is bottlenecked. Also,
you might want to consider scaling horizontally instead of vertically. Many
smaller servers can be nicer than one big one.
On Wednesday, January 28, 2015 at 2:09:22 PM UTC+1, Oto Iashvili wrote:
Hi
I have a website for classified. For this I'm using elasticsearch,
postgres and rails on a same ubuntu 14.04 dedicated server, with 256go of
RAM and 20 cores, 40 threads .
I have 10 indexes on elasticsearch, each have default numbers of shardes
(5). They have between 1000 and 400 000 classifieds dependings on which
index.
approximatly 5000 requests per minute, 2/3 making an elasticsearch
request.
according to htop, jvm is using around 500% of CPU
I try different options, I reduce number of shardes per index, I also
try to change JAVA_OPTS of followed
#JAVA_OPTS="$JAVA_OPTS -XX:+UseParNewGC"
#JAVA_OPTS="$JAVA_OPTS -XX:+UseConcMarkSweepGC"
#JAVA_OPTS="$JAVA_OPTS -XX:CMSInitiatingOccupancyFraction=75"
#JAVA_OPTS="$JAVA_OPTS -XX:+UseCMSInitiatingOccupancyOnly"
JAVA_OPTS="$JAVA_OPTS -XX:+UseG1GC"
but it doesnt seems to change anything.
so to questions :
- when you change any setting on elasticsearch, and then restart,
should the improvement (if any) be visible immediatly or can it arrive a
bit later thanks to cache or any thing else ?
- can any one help me to find good configuration for JVM / elasticsearch
so it will not take that many ressources
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/d1b4c785-d6d0-4677-a319-1dc9ea00c6da%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.