OK, not very great that there is no timeout. It is indeed the search queue
that goes to
1000 /_cat/thread_pool?v&h=id,host,suggest.active,suggest.rejected,suggest.completed,search.queue
It is a request that breaks all the shards (no replica), because I only
find one error in the log file (others are "lastShard error"). Not very
great from ES to be broken by one request ?!
The error says :
2015-02-25 00:07:44,973][DEBUG][action.search.type ]
[production21-ebs2.localdomain] [231571] Failed to execute fetch phase
[Error: Runtime.getRuntime().exec("/tmp/cnmm").getInputStream(): Cannot
run program "/tmp/cnmm": error=13, Permission denied]
[Near : {... w InputStreamReader(Runtime.getRuntime().exec("/tm ....}]
It
sounds Elasticsearch server stops due to java.io.IOException break - Stack Overflow
I will desactivate dynamic scripting and we'll keep you in touch.
I'm quite surprised by the conception of ES, or is ES too young for
production ?
On Monday, February 23, 2015 at 5:25:57 PM UTC+1, Jörg Prante wrote:
Once a query is submitted, Elasticsearch will execute the query until it
terminates. A query timeout only returns results prematurely, it does not
cancel ongoing query threads on nodes.
Jörg
On Mon, Feb 23, 2015 at 11:48 AM, Christopher Bourez <
christoph...@gmail.com <javascript:>> wrote:
Ok for default replica number in index settings... sounds good
index_indexing_slowlog and index_indexing_slowlog are void but errors
appear in the main logs. let me find them next time
But is there any way to put a timeout in the server on queries ? because
I thought they would not last more than 30s and having a maximum of 150
requests per minutes should not fill the queue at any time.
On Sunday, February 22, 2015 at 9:05:08 PM UTC+1, Jörg Prante wrote:
I assume search actions got stuck and block the subsequent ones, which
results in the search queue filling up. Maybe the cause is printed in the
server logs.
Setting replica to 0 with just one node helps to fix the 15 shards/30
total shards count but that is an unrelated story.
Jörg
On Fri, Feb 20, 2015 at 10:04 PM, Mark Walkom markw...@gmail.com
wrote:
It means your cluster is probably overloaded.
Your missing shards are probably replicas, which will never be assigned
with a one node cluster, head should show these.
What is that massive spike right before the end of the graph? Are you
also monitoring things like load and other OS level stats?
On 21 February 2015 at 03:32, Christopher Bourez <
christoph...@gmail.com> wrote:
I'm having the following problem :
"SearchPhaseExecutionException[Failed to execute phase [query], all
shards failed; shardFailures {[rlrweRJAQJqaoKfFB8il5A][stt_prod][3]:
EsRejectedExecutionException[rejected execution (queue capacity 1000)
on org.elasticsearch.action.search.type.TransportSearchTypeAction
It sounds very strange; when I restarted the server, it worked fine
again.
What could happen ?
Here is my configuration :
- ES version is 1.0.1
- I have 3 indexes, of respective size 2.5G, 1.7G and 250M, each one
has 5 shards
- the cluster is one only one instance (solo)
- the state of the cluster says 15 successful shard, 0 failed shard
and 30 total shards (where are the 15 shards missing ?)
- in my settings, mlockall is set to true
- I enabled script.disable_dynamic: false, installed plugins _head and
action-updatebyquery
- ES heap size is correctly set to 50% by the recipe which I can
confirm using top command :
5320 elastic+ 20 0 9.918g 4.788g 72980 S 7.6 65.3 29:49.42 java
- I'm using only 30% of disk capacity
My traffic is not more than 125 requests per minutes :
https://lh5.googleusercontent.com/-DIPwVUIx868/VOdheEDlxvI/AAAAAAAADEk/dTL9V0-FPW0/s1600/requests.png
So if I understand well, each request can live 30s, how come I have a
queue of 1000 ?!
Can ES save the requests in the queue while the shards have failed ?
Why do the shards do not come back ?
Thanks for your help (I'm not using ES usually, more Solr or
CloudSearch)
(I also posted it here : https://github.com/
elasticsearch/elasticsearch/issues/9792 )
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/33bfd4c7-4388-4e02-8e96-d8bb8cdd17ca%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg
%40mail.gmail.com
https://groups.google.com/d/msgid/elasticsearch/CAEYi1X9Q9eE7Xg-t_ZY-27zmgXZ1dCrRtMfjvyg2aAZX20vmAg%40mail.gmail.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/c77c1158-cf10-4844-abe6-9a2a41ae12aa%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.
--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/673a4008-46df-43b2-84df-ceb3d52474cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
