Hi there,
Recently i have built elasticsearch with fluentd and Kibana and all is working fine, but we are facing high cpu load while performing search caused by java.
I have 60 GB Ram and 16 processors and each processor has 2 cores.
ES_HEAP_SIZE=32g
-find below more details
curl "localhost:9200/_cat/thread_pool?v&h=search.rejected"
search.rejected
387
curl "localhost:9200/_cat/thread_pool?v&h=index.rejected"
index.rejected
0
- snapshot from elasticsearch.log
[DEBUG][action.search.type ] [Hulk 2099] [logstash-2015.03.14][4], node[qxcAN3lURs65Lf1GMhB_qg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@7c71025f] lastShard [true]
org.elasticsearch.common.util.concurrent.EsRejectedExecutionException: rejected execution (queue capacity 1000) on org.elasticsearch.search.action.SearchServiceTransportAction$23@1d7c9f0f
at org.elasticsearch.common.util.concurrent.EsAbortPolicy.rejectedExecution(EsAbortPolicy.java:62)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:821)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1372)
at org.elasticsearch.search.action.SearchServiceTransportAction.execute(SearchServiceTransportAction.java:551)
at org.elasticsearch.search.action.SearchServiceTransportAction.sendExecuteQuery(SearchServiceTransportAction.java:228)
at org.elasticsearch.action.search.type.TransportSearchCountAction$AsyncAction.sendExecuteFirstPhase(TransportSearchCountAction.java:71)
at org.elasticsearch.action.search.type.TransportSearchTypeAction$BaseAsyncAction.performFirstPhase(TransportSearchTypeAction.java:171)
Now i read that i have to update the threadpool settings in ES and i have now
curl -XGET localhost:9200/_cluster/settings?pretty
{
"persistent" : { },
"transient" : { }
}
i'm trying to update the settings with the below commands
curl -XPUT localhost:9200/_cluster/settings -d '{
"threadpool" : {
"index": {
"type": "fixed",
"size": 32,
"queue_size": 1000
},
"bulk": {
"type": "fixed",
"size": 32,
"queue_size": 1000
},
"search": {
"type": "fixed",
"size": 96,
"queue_size": 1000
}
}
}'
but i keep getting {"error":"ActionRequestValidationException[Validation Failed: 1: no settings to update;]","status":400}
What's wrong with this command? is it the right solution for my issue ?
Please advise
Thanks.
Ayman