Ransom attack on Elasticsearch cluster?

No sensitive data on our test cluster.

Anyway this post is to remind others to be careful on their ES security :slight_smile:

Yeah. Thanks for sharing this. That can help others for sure.

Fortunately no one probably:

  • uses elasticsearch as the single source of truth for the data (single datastore)
  • exposes critical data directly on internet
  • with no security
  • with no backup

Well... let's hope that is true...

You can check on public IoT search engines like https://www.shodan.io/

Last time I looked there were thousands of open Elasticsearch servers reported. People never learn.

1 Like

For anyone interested, I wrote a blog post on how your clusters can and should be protected: http://code972.com/blog/2017/01/107-dont-be-ransacked-securing-your-elasticsearch-cluster-properly

2 Likes

just curious, anyone know how does they hack? use script in query DSL?

If the cluster is open to the internet and not secured in any way, you can simply read/modify/delete the data through the REST interface. No need for scripts of any kind.

Yup, and then probably port scanning or using tools like https://www.shodan.io

Our ES cluster is also hacked . All the indices were removed and a warning message is shown . please suggest some security measures .

Thanks

Read https://www.elastic.co/blog/protecting-against-attacks-that-hold-your-data-for-ransom

It's a shame that Elastic thinks data security is optional and doesn't include security modules as part of the default bundle. I understand that as an commercial entity, they have to generate revenue and shield is one of them. But most free to use products do not consider security as such. Before anyone says, stop cribbing, it's a free product, blah, blah and pay for it already, I will shut up :slight_smile:

4 Likes

Shield is now replaced by X-Pack. But I cannot understand if it is free or I have to pay for it. Does anyone know something more about X-Pack?

The Security part of X-Pack is a commercial offering.

Here's the problem:

  1. User finds out ES doesn't do authentication out of the box.
  2. User googles ES auth, finds X-Pack
  3. User installs X-Pack, auth starts to work.
  4. Trial license expires, user gets a free basic license
  5. Warning states that "some features" are subject to payment and not included in the basic license.
  6. ES is public again.

I appreciate that you guys put in an acknowledge mechanism so users have to ack that some features are not included in the basic license.

I do not appreciate that you do not mention clearly in your documentation that this pertains ALL AUTHENTICATION.

Here's what the documentation states:

https://www.elastic.co/guide/en/x-pack/current/license-expiration.html

# License [will expire] on [Friday, December 30, 2016]. If you have a new license, please update it.
# Otherwise, please reach out to your support contact.
#
# Commercial plugins operate with reduced functionality on license expiration:
# - security
# -- Cluster health, cluster stats and indices stats operations are blocked
# -- All data operations (read and write) continue to work
# - watcher
# -- PUT / GET watch APIs are disabled, DELETE watch API continues to work
# -- Watches execute and write to the history
# -- The actions of the watches don't execute
# - monitoring
# -- The agent will stop collecting cluster and indices metrics
# -- The agent will stop automatically cleaning indices older than [xpack.monitoring.history.duration]
# - graph
# -- Graph explore APIs are disabled

Notice the indentation underneath "- security"? (I added two dashes because the post formatter eats the extra space)
This indentation means "security" is listed as a category of items to be listed underneath. The two items ("Cluster health, cluster stats and indices stats operations are blocked", "All data operations (read and write) continue to work") do NOT adequately transport the message that this main feature of X-Pack is actually not working anymore.

Here's how you fix this: Either

a) Put up a big notice that makes it absolutely clear that authentication is not included in the free basic license, or
b) Consider adding auth to the free license tier. It is not something you can reasonably expect to charge money for. It is easily achieved via other means, so why not aim for robustness instead and offer everyone the benefits of a basic functionality like this. You may think it's a smart business idea, but to people outside your company it just looks like a cash grab.

We've been hit by this as well, but our data was easily reindexed. However, this just makes ES look immature. This would be an awesome time to turn this into a PR advantage, and react by adding auth support to either the free x-pack license, or even just include it in the main program.

2 Likes

My client has been hacked too, I'm reading your guide, thx Itamar!

  • Reverse proxy to ES and do authentication there.
  • Setup firewall rules to only allow traffic from trusted IP addresses.
  • Tunnel via SSH
  • ...?

I don't see how anyone could miss that ES doesn't authenticate requests by itself. If you've put an ES endpoint on the public internet you only have yourself to blame when someone else accesses it.

I'm suspicious of your conclusion that licence expiry disables all authentication. It certainly doesn't seem to here, but I'm a little outdated in versions.

dmjcgreal: Indeed, anyone putting up an ES instance without authentication on a public IP has to face the risk of someone else deleting stuff from it.

But do go ahead and try the latest version, install X-Pack (auth will work then), then install a basic license. You will have to acknowledge but it is easy to miss.

Thankfully, several solutions exist: There's an open source plugin for authentication which does not cost money, and you can also use nginx or haproxy to add basic auth to an ES instance. In general it's safest to not expose ES at all if you can help it.

And again, while there is a page buried somewhere that lists authentication as a paid feature, I would be helpful to include this information in the documentation as well - because that's the page you get if you google ES auth.

There are also free security solutions like https://github.com/sscarduzio/elasticsearch-readonlyrest-plugin or https://github.com/floragunncom/search-guard

Having security optional is the worst thing elastic managers came up with.

We've been driven away and have moved our largest installation back to MongoDB, where it was months ago before we moved to elastic. Luckily its all plug and play, so that one was a matter of hours. The other products will follow up.

I suggest you reconsider the x-pack licensing asap.

nkoothrappahli You are correct and thanks for linking these here in case others google for this issue.

I still hope Elastic can rethink their licensing approach and what parts of their X-Pack product they consider to be something you need to pay for. Especially since it's difficult to argue that adding basic auth is such a tremendous feat of engineering.

Alright, I have a huge cluster with billions of documents. It scares me to death every time I see post like this which unfortunately the number of blogs and reports are growing recently.

The simplest way is to have cluster in private IP behind reverse_proxy such as Nginx with a simple authentication. Only expose the Nginx to public and disable DELETE or even PUT/POST over your selected endpoints/users. Even install a simple fail2ban on the machine to ban IP addresses that try to brute force your simple authentication. (this comes integrated with Nginx)

Is this so hard to do or this is not good enough that's why nobody even bothers to do it?

I get the criticism over x-pack that the Security should be shipped with Elastic Stack. Maybe they do this in the future. Or maybe they won't, but by no means this justifies thousands of MongoDB and Elasticsearch being exposed publicly with the same ports!

My 2 cents, use Nginx! It is simple, tons of snippets out there to get you up and running if you've never done reverse proxy. Besides, it is very useful and light weight.

May the force be with our Elastic Stack!

3 Likes