Where did these indices come from?

Elastic Stack Elasticsearch
1

I have Elasticsearch running on two nodes with kibana on one of them. I have port 9200 closed on the firewall and can't access it from outside the network. When I list the indices, it shows several that I did not create. It looks like .kibana is legitimate, but I am worried about the other ones. Are these indices supposed to be there, or is this a sign that the servers have been compromised?

root@web-server-03:/var/log/elasticsearch# curl 'localhost:9200/_cat/indices?v'
health status index                      pri rep docs.count docs.deleted store.size pri.store.size 
green  open   mambo                        5   1          0            0      1.3kb           720b 
green  open   newsletter                   5   1          0            0      1.3kb           720b 
green  open   modules                      5   1          0            0      1.3kb           720b 
green  open   robohelp                     5   1          0            0      1.3kb           720b 
green  open   create_account_process.php   5   1          0            0      1.2kb           720b 
green  open   action.php                   5   1          0            0      1.2kb           720b 
green  open   index.htm                    5   1          0            0      1.3kb           720b 
green  open   cgi-bin                      5   1          0            0      1.2kb           720b 
green  open   login.jsp                    5   1          0            0      1.3kb           720b 
green  open   reports                      5   1          0            0      1.3kb           720b 
green  open   admin                        5   1          0            0      1.3kb           720b 
green  open   modules.php                  5   1          0            0      1.3kb           720b 
green  open   xmlrpc.php                   5   1          0            0      1.3kb           720b 
green  open   payonline.asp                5   1          0            0      1.3kb           720b 
green  open   forum                        5   1          0            0      1.3kb           720b 
green  open   nquser.php                   5   1          0            0      1.2kb           720b 
green  open   flatnuke                     5   1          0            0      1.3kb           720b 
green  open   server.php                   5   1          0            0      1.3kb           720b 
green  open   netquery                     5   1          0            0      1.2kb           720b 
green  open   e107                         5   1          0            0      1.3kb           720b 
green  open   auth.php                     5   1          0            0      1.3kb           720b 
green  open   weblibs.pl                   5   1          0            0      1.2kb           720b 
green  open   jhot.php                     5   1          0            0      1.3kb           720b 
green  open   tiki                         5   1          0            0      1.3kb           720b 
green  open   tikiwiki                     5   1          0            0      1.3kb           720b 
green  open   goform                       5   1          0            0      1.3kb           720b 
green  open   login.php                    5   1          0            0      1.3kb           720b 
green  open   check                        5   1          0            0      1.3kb           720b 
green  open   index.php                    5   1          0            0      1.3kb           720b 
green  open   users                        5   1          0            0      1.3kb           720b 
green  open   shopproductselect.asp        5   1          0            0      1.3kb           720b 
green  open   data                         5   1          0            0      1.3kb           720b 
green  open   .kibana                      1   1         14            1     52.3kb         26.1kb 
green  open   search.php                   5   1          0            0      1.2kb           720b 
green  open   shopping                     5   1          0            0      1.2kb           720b 
green  open   blog                         5   1          0            0      1.3kb           720b
2

I seen something similar even though my elasticsearch cluster is protected by firewall. Do you have a Qualys vulnerability scanner? I suspect in my case it is qualys that is creating the fake indices as a part of scanner for vulnerabilities

3

You do have you cluster firewalled off right?
Also, disable auto creation of indices, that will help.

4

Check your reverse proxy that forwards HTTP requests. It does not filter illegal requests. You receive messages from "script kids" who try to break into broken PHP service.

5

oh no dude.. that aint good.. you can check by telnet to each one of elasticsearch nodes on the port and make sure only the node you authorized able to access precious es resources.. you can use command nc or telnet.

hth

6

That would make sense, I do have AlertLogic performing vulnerability scans every week (from outside of the network, and from within). looking at the logs, it is reporting a few vulnerabilities on port 9200. One of which is related to modules.php. I don't see anything in the logs or docs regarding the other indices though. I thought I had it limited to localhost, and the two server IP's though. I'll have to double-check on that.

7

Yea, it is firewalled off. I'll go ahead and disable auto creation, but I'm more worried about someone being able to access the data, than creating random indices.

8

Just verified that 9200 is indeed closed on both server. I suspect it is coming from our vulnerability scanner. Checking with the provider right now to confirm.

9

What should I be checking on the reverse proxy? The only one I have on those servers is for Kibana:

<VirtualHost *:80>
        ServerName kibana.[mydomain].com
        ErrorLog ${APACHE_LOG_DIR}/kibana.[mydomain].com-error.log            
        CustomLog ${APACHE_LOG_DIR}/kibana.[mydomain].com-access.log combined
        ProxyRequests Off
        ProxyPass / http://localhost:5601/
        ProxyPassReverse / http://localhost:5601/
    <Location />
      Order deny,allow
      Deny from all
      Allow from [my.ip]
    </Location>
</VirtualHost>
10

The proxy allows unlimited access and is wide open

You allow

  • HTTP POST and PUT on all Elasticsearch operations
  • HTTP DELETE which can potentially remove all of your data

You should at least restrict the proxy to access _search endpoint only.

Check your clients how they create indices (either by explicit index creation command or by sending documents) and maybe you can disable automatic index creation in Elasticsearch configuration when indexing documents. I recommend this strongly:

action.auto_create_index: 0

11

So I did get AlertLogic to confirm that their scan most likely did create those indices. I deleted all of them and set auto_create_index to 0 on both servers, but they came right back as soon as the next scan ran. I could probably restrict the scan from trying port 6200, but that seems like an incomplete fix. I'll try making Elasticsearch only respond to the servers I want them to communicate with.

@jprante I'm not really sure what you mean, where do see that my proxy is wide open? Isn't that open on just the 5601 port, and only from my ip address?

12

I'd try action.auto_create_index: false as per the docs.

13

Ok, changed it to false and deleted the new indices. I'll let you know if they pop back up on the next scan.

Thanks guys.

14

The proxy configuration you show allows every request to get through like a port forwarder. It does not protect against (harmful) HTTP PUT or HTTP DELETE. Also it does not restrict access to endpoints of Elasticsearch like _search so the proxy does not protect Elasticsearch from delivering cluster state or other (maybe sensitive) information.

15

Updated the proxy configuration:

<VirtualHost *:80>
    ServerName kibana.[mydomain].com
    ErrorLog ${APACHE_LOG_DIR}/kibana.[mydomain].com-error.log
    CustomLog ${APACHE_LOG_DIR}/kibana.[mydomain].com-access.log combined
    ProxyRequests Off
    ProxyPass / http://localhost:5601/
    ProxyPassReverse / http://localhost:5601/
    <Location />
        Order deny,allow
        Deny from all
        Allow from [my.ip]

        <Limit GET >
            order deny,allow
            deny from all
            allow from 127.0.0.1
            allow from [my.ip]
        </Limit>

        <Limit POST PUT DELETE>
            order deny,allow
            deny from all
        </Limit>
        AuthName "Access Restricted"
        AuthType Basic
        AuthUserFile "/etc/apache2/.htpasswd"
        Require valid-user
    </location>
</VirtualHost>