Artifacts.elastic.co intermittently fails (404, "no longer has a Release file.")

Elastic Stack Elasticsearch
1

It seems that there are transient failures with artifacts.elastic.co, perhaps servers that are out of sync. We sometimes see this when running apt-get -q update:

bad

$ sudo apt-get -q update
...
Ign:5 https://artifacts.elastic.co/packages/5.x/apt stable InRelease
Err:6 https://artifacts.elastic.co/packages/5.x/apt stable Release
  404  Not Found [IP: 34.120.127.130 443]
...
Reading package lists...
E: The repository 'https://artifacts.elastic.co/packages/5.x/apt stable Release' no longer has a Release file.

good

$ sudo apt-get -q update
...
Ign:8 https://artifacts.elastic.co/packages/5.x/apt stable InRelease
Hit:9 https://artifacts.elastic.co/packages/5.x/apt stable Release

This can be reproduced with curl:

bad

$ curl --silent -D - https://artifacts.elastic.co/packages/5.x/apt/dists/stable/Release
HTTP/2 404 
server: nginx/1.17.8
date: Wed, 03 Mar 2021 19:07:19 GMT
content-length: 9
strict-transport-security: max-age=15724800; includeSubDomains
via: 1.1 google
age: 20
alt-svc: clear

good

HTTP/2 200 
server: nginx/1.17.8
date: Wed, 03 Mar 2021 19:07:00 GMT
content-type: application/octet-stream
content-length: 5116
last-modified: Tue, 19 Mar 2019 14:52:46 GMT
etag: "f0c208efc3f994f168faa64cb26730e2"
accept-ranges: bytes
strict-transport-security: max-age=15724800; includeSubDomains
via: 1.1 google
age: 52
cache-control: public, max-age=120
alt-svc: clear

Origin: elastic
Label: . stable
Suite: stable
Codename: stable
Date: Tue, 19 Mar 2019 14:51:51 UTC
Architectures: amd64 i386
Components: main
Description: Generated by aptly
MD5Sum:
 5fda4d0ec4ff29c476c1514390a778d8   635567 Contents-amd64.gz
 5fda4d0ec4ff29c476c1514390a778d8   635567 Contents-i386.gz
...

I have enough hosts that I was able to make sure it wasn't a version of apt or anything, and curl shows that it isn't apt-specific either. I tested out versions of tls1.[0123] and http1.1/http2.0 to ensure it wasn't a difference in those. Nope.

Another thought was that it had to do with the age of the elastic.co packages (5.x). However, two things disprove this. First, I've been able to run curl and hit a bad host with each of the three versions ([567].x). Second, it's been reported by others on those versions too.

6.x on discuss, Oct 2020, no reply
7.x on discuss, Jan 2021, no reply
5.x and 6.x on a closed github issue, Oct 2020-Feb 2021. Two complaints (plus a third as a +1), no response.

I'll link to this thread from the github issue. Even if this one doesn't get a reply, hopefully it'll help others who run into this issue and can't figure out why- TLDR: elastic.co has a problem.

1 Like
2

Thanks for adding the additional data into this, it's useful to help identify the issue.
We're taking a closer look and will hopefully have a response for you soon.

1 Like
3

Any updates on this, Mark? I haven't seen errors from our systems, which makes me think it's been handled (or we just haven't gotten unlucky enough to hit it).

4

We made some changes to the way our CDN works, which we believe will make it more compatible with apt.
It's good to hear that the change seems to be working - please let us know if the problem comes back.

1 Like
5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.