Official apt repository (451 Unavailable For Legal Reasons)

First off, sorry for the continued trouble. We've been playing whack-a-mole here and need to get this properly fixed with our CDN provider.

@MBConsultingUK, is there a public list of DO's IP ranges? I could only find a statement that there is no such list.


If anyone is blocked, you could try the following. Just note that this is a hack and only one possible approach:

I'm on Ubuntu 18.04 and let's assume I have added the Elastic DEB repository:

wget -qO - https://artifacts.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
echo "deb https://artifacts.elastic.co/packages/7.x/apt stable main" | sudo tee -a /etc/apt/sources.list.d/elastic-7.x.list

Now fetching the binary fails with the 451 response code. But I have uploaded the binary to my DO Spaces.

Then I just install the DEB manually and ideally compare the hash, start Elasticsearch, and check that it's working as expected:

wget https://xeraa-artifacts.ams3.digitaloceanspaces.com/elasticsearch-7.7.1-amd64.deb
shasum -a 512 elasticsearch-7.7.1-amd64.deb
sudo dpkg -i elasticsearch-7.7.1-amd64.deb 
sudo service elasticsearch start
curl localhost:9200

Output:

{
  "name" : "ubuntu-s-1vcpu-2gb-ams3-01",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "7arXpqcdSMSRLglhqJ6JAw",
  "version" : {
    "number" : "7.7.1",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "ad56dce891c901a492bb1ee393f12dfff473a423",
    "build_date" : "2020-05-28T16:30:01.040088Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Now let's assume we have fixed the 451 problem and you want to install the current version. This will just work, so the manual installation of the DEB shouldn't block you further down the road (including the full output here for clarity):

sudo apt-get update && sudo apt-get upgrade
Hit:1 https://artifacts.elastic.co/packages/7.x/apt stable InRelease
Get:2 http://mirrors.digitalocean.com/ubuntu bionic InRelease [242 kB]                                
Hit:3 http://mirrors.digitalocean.com/ubuntu bionic-updates InRelease                                           
Hit:4 http://security.ubuntu.com/ubuntu bionic-security InRelease       
Hit:5 http://mirrors.digitalocean.com/ubuntu bionic-backports InRelease 
Fetched 242 kB in 0s (501 kB/s)                   
Reading package lists... Done
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  grub-pc-bin libdumbnet1
Use 'sudo apt autoremove' to remove them.
The following packages have been kept back:
  linux-headers-generic linux-headers-virtual linux-image-virtual linux-virtual netplan.io
The following packages will be upgraded:
  elasticsearch
1 upgraded, 0 newly installed, 0 to remove and 5 not upgraded.
Need to get 319 MB of archives.
After this operation, 5225 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
Get:1 https://artifacts.elastic.co/packages/7.x/apt stable/main amd64 elasticsearch amd64 7.8.0 [319 MB]
Fetched 319 MB in 5s (65.1 MB/s)        
(Reading database ... 61082 files and directories currently installed.)
Preparing to unpack .../elasticsearch_7.8.0_amd64.deb ...
Unpacking elasticsearch (7.8.0) over (7.7.1) ...
Setting up elasticsearch (7.8.0) ...
Created elasticsearch keystore in /etc/elasticsearch/elasticsearch.keystore
Processing triggers for ureadahead (0.100.0-21) ...
Processing triggers for systemd (237-3ubuntu10.41) ...

sudo service elasticsearch restart

curl localhost:9200
{
  "name" : "ubuntu-s-1vcpu-2gb-ams3-01",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "7arXpqcdSMSRLglhqJ6JAw",
  "version" : {
    "number" : "7.8.0",
    "build_flavor" : "default",
    "build_type" : "deb",
    "build_hash" : "757314695644ea9a1dc2fecd26d1a43856725e65",
    "build_date" : "2020-06-14T19:35:50.234439Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Hope that helps you move forward if you are really stuck; it should also be less invasive than setting up a VPN just for this and relatively simple.

Hey @xeraa - I'm not sure about the list of ranges, but I'll ask the question.

I'm in the UK and most (all?) of the DO support team seem to be in the US timezone, so it could be a while before I get a result though.

Thanks for the workaround, I'll give it a go when I'm back at work tomorrow.

having the same issue with digitalocean.
we have two instances there: 188.226.177.X is HTTP 451 while 188.226.165.Y is okay.
how can this be an issue after months of initial reports?

Still happening here. Connecting from digital ocean AMS3 !

@xeraa A list of all of DigitalOcean's IPs is available here: http://digitalocean.com/geo/google.csv

This is provided for GEOIP databases and Google, but it should be useful for this whitelisting use-case.

@MBConsultingUK FWIW, DO is a 24/7 team. We have people working in all timezones to answer support tickets.

1 Like

@ErisRenee yeah, apologies, that came across as kinda brutal on my part, I tend to hang out in the Partner Slack channels where it's very light on content until the US folks come online, I should have clarified that was what I meant.

Sorry if any offence was caused :slight_smile:

I downloaded the file manually, in my case I got:

E: Failed to fetch https://artifacts.elastic.co/packages/7.x/apt/pool/main/e/elasticsearch/elasticsearch-7.8.0-amd64.deb  451  Unavailable For Legal Reasons [IP: 151.101.14.222 443]

So I downloaded https://artifacts.elastic.co/packages/7.x/apt/pool/main/e/elasticsearch/elasticsearch-7.8.0-amd64.deb manually to my PC, and uploaded the file to the server, then ran gdebi elasticsearch-7.8.0-amd64.deb to install it manually.

more DO droplets are experiencing this issue. And they were previously working fine

The issue seems to be resolved for me today, a week after the server went live, hopefully I can smoothly move it over to filebeat without it sending too many logs at once.

Can confirm that this is now working for me as well

Thanks for everybody's patience! Our provider told us that they are rolling out a more comprehensive upgrade over the weekend. If anybody is still having issues, please share your IP and region (if possible). I really hope that this is fixed now...

Hey everyone! I just wanted to provide a note to (hopefully) wrap up this issue.

As of a few days ago, we've received word from our CDN provider that the underlying cause for these IP range mismatches should be resolved. We took some steps to independently verify that this is indeed the case, and according to the metrics we've collected, the denials stemming from 451 responses have fallen off dramatically to levels that look normal once again and not denying a large share of traffic that was previously being denied.

At the time of this writing, as far as we're aware, the behavior from these URLs should be normal again. If someone hits these response codes from an unexpected location again, please do let us know - we don't want to impede access and create problems where there should be access.

I apologize for any inconvenience this caused - although we do need to have some controls in place for certain countries, this did end up being caused by some underlying incorrect IP geolocation data, which isn't the fault of users or operators within those IP ranges that experienced issues. We'll try to avoid this situation in the future!

Hi Tyler, I'm still having the issue from 38.117.105.228

Thanks @hitman247m, I'll follow up with the CDN provider for that address.