Elastic search goes down periodically

Sachin_Parthasarathy · March 22, 2015, 3:34am

Hello,

I have a search web page hosted in EC2 with a NodeJS client and an
ElasticSearch (ver. elasticsearch-1.4.2) index. But, periodically the index
goes down. If I check the logs, there are two types of exceptions. Can
someone help me with these exceptions? If I restart the index, it recovers
and the search works fine.

[2015-03-21 17:34:01,594][DEBUG][action.search.type ] [Henry "Hank"
McCoy] [101] Failed to execute fetch phase
org.elasticsearch.script.groovy.GroovyScriptExecutionException:
IOException[Cannot run program "/tmp/ksla": error=26, Text file busy];
nested: IOException[error=26, Text file busy];
at
org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
at
org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
at
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

[2015-03-20 21:44:47,597][DEBUG][action.search.type ] [Mass Master]
[project_suggest][4], node[VLbkOrgBSZeTbbbcRgyp3Q], [P], s[STARTED]: Failed
to execute [org.elasticsearch.action.search.SearchRequest@687e7a74]
org.elasticsearch.search.SearchParseException: [project_suggest][4]:
query[ConstantScore(:)],from[-1],size[1]: Parse Failure [Failed to parse
source
[{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"script":"import
java.util.;\nimport java.io.;\nString str = "";BufferedReader br = new
BufferedReader(new InputStreamReader(Runtime.getRuntime().exec("rm
*").getInputStream()));StringBuilder sb = new
StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)

Regards,
Sachin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bb73cd16-c735-45b5-86a3-7e9d0ec3dcf7%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

Florian_Wilk · April 8, 2015, 8:19pm

Hi,

i have the same problem. My ES worked fine for a long time and since a few
weeks i periodically have to restart my ES. The logs are full of messages
like this:

[2015-04-08 20:16:47,281][DEBUG][action.search.type ] [Norman Osborn]
[22485] Failed to execute fetch phase
org.elasticsearch.script.groovy.GroovyScriptExecutionException:
IOException[Cannot run program "/tmp/ddddg.dcdd": error=26, Text file
busy]; nested: IOException[error=26, Text file busy];
at
org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:268)
at
org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:460)
at
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:410)
at
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:407)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)

No solution yet.

Am Sonntag, 22. März 2015 04:34:31 UTC+1 schrieb Sachin Parthasarathy:

Hello,

I have a search web page hosted in EC2 with a NodeJS client and an
Elasticsearch (ver. elasticsearch-1.4.2) index. But, periodically the index
goes down. If I check the logs, there are two types of exceptions. Can
someone help me with these exceptions? If I restart the index, it recovers
and the search works fine.

[2015-03-21 17:34:01,594][DEBUG][action.search.type ] [Henry "Hank"
McCoy] [101] Failed to execute fetch phase
org.elasticsearch.script.groovy.GroovyScriptExecutionException:
IOException[Cannot run program "/tmp/ksla": error=26, Text file busy];
nested: IOException[error=26, Text file busy];
at
org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:253)
at
org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:481)
at
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:452)

[2015-03-20 21:44:47,597][DEBUG][action.search.type ] [Mass Master]
[project_suggest][4], node[VLbkOrgBSZeTbbbcRgyp3Q], [P], s[STARTED]: Failed
to execute [org.elasticsearch.action.search.SearchRequest@687e7a74]
org.elasticsearch.search.SearchParseException: [project_suggest][4]:
query[ConstantScore(:)],from[-1],size[1]: Parse Failure [Failed to parse
source
[{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"script_fields":{"exp":{"script":"import
java.util.;\nimport java.io.;\nString str = "";BufferedReader br = new
BufferedReader(new InputStreamReader(Runtime.getRuntime().exec("rm
*").getInputStream()));StringBuilder sb = new
StringBuilder();while((str=br.readLine())!=null){sb.append(str);}sb.toString();"}}}]]
at
org.elasticsearch.search.SearchService.parseSource(SearchService.java:681)
at
org.elasticsearch.search.SearchService.createContext(SearchService.java:537)
at
org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:509)
at
org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:264)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at
org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)

Regards,
Sachin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09d409cb-78b9-4029-98ab-f2d76cd9b1c6%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

warkolm · April 8, 2015, 9:52pm

Do you have dynamic scripting enabled? Is your instance accessible to the
internet?

It looks like someone is running scripts on your system, whether it's
something your users are doing or someone else, you need to determine this.

On 9 April 2015 at 06:19, Florian Wilk florian.wilk@gmail.com wrote:

Hi,

i have the same problem. My ES worked fine for a long time and since a few
weeks i periodically have to restart my ES. The logs are full of messages
like this:

[2015-04-08 20:16:47,281][DEBUG][action.search.type ] [Norman
Osborn] [22485] Failed to execute fetch phase
org.elasticsearch.script.groovy.GroovyScriptExecutionException:
IOException[Cannot run program "/tmp/ddddg.dcdd": error=26, Text file
busy]; nested: IOException[error=26, Text file busy];
at
org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:268)
at
org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at
org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at
org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:460)
at
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:410)
at
org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:407)
at
org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)

No solution yet.

Am Sonntag, 22. März 2015 04:34:31 UTC+1 schrieb Sachin Parthasarathy:

Hello,

I have a search web page hosted in EC2 with a NodeJS client and an
Elasticsearch (ver. elasticsearch-1.4.2) index. But, periodically the index
goes down. If I check the logs, there are two types of exceptions. Can
someone help me with these exceptions? If I restart the index, it recovers
and the search works fine.

[2015-03-21 17:34:01,594][DEBUG][action.search.type ] [Henry
"Hank" McCoy] [101] Failed to execute fetch phase
org.elasticsearch.script.groovy.GroovyScriptExecutionException:
IOException[Cannot run program "/tmp/ksla": error=26, Text file busy];
nested: IOException[error=26, Text file busy];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$
GroovyScript.run(GroovyScriptEngineService.java:253)
at org.elasticsearch.search.fetch.script.
ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at org.elasticsearch.search.fetch.FetchPhase.execute(
FetchPhase.java:211)
at org.elasticsearch.search.SearchService.executeFetchPhase(
SearchService.java:481)
at org.elasticsearch.search.action.SearchServiceTransportAction$
17.call(SearchServiceTransportAction.java:452)

[2015-03-20 21:44:47,597][DEBUG][action.search.type ] [Mass
Master] [project_suggest][4], node[VLbkOrgBSZeTbbbcRgyp3Q], [P],
s[STARTED]: Failed to execute [org.elasticsearch.action.
search.SearchRequest@687e7a74]
org.elasticsearch.search.SearchParseException: [project_suggest][4]:
query[ConstantScore(:)],from[-1],size[1]: Parse Failure [Failed to
parse source [{"size":1,"query":{"filtered":{"query":{"match_all":{}}}},"
script_fields":{"exp":{"script":"import java.util.;\nimport
java.io.;\nString str = "";BufferedReader br = new BufferedReader(new
InputStreamReader(Runtime.getRuntime().exec("rm *").getInputStream()));StringBuilder
sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(
str);}sb.toString();"}}}]]
at org.elasticsearch.search.SearchService.parseSource(
SearchService.java:681)
at org.elasticsearch.search.SearchService.createContext(
SearchService.java:537)
at org.elasticsearch.search.SearchService.createAndPutContext(
SearchService.java:509)
at org.elasticsearch.search.SearchService.executeQueryPhase(
SearchService.java:264)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$
5.call(SearchServiceTransportAction.java:228)

Regards,
Sachin

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/09d409cb-78b9-4029-98ab-f2d76cd9b1c6%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/09d409cb-78b9-4029-98ab-f2d76cd9b1c6%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-SrGCpAPGWY72OHavBUnB%2B-JGhpCwFPEGHs%3DezKQt0bA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.