Possible Faulty CDN Node?


(Tyler Langlois) #1

We've been hearing some reports that certain images aren't being served to some users over the CDN Discourse uses. For example:

Some initial investigation led to finding this image being served over p-uswd00.kxcdn.com (107.182.231.101) as opposed to some other (working) CDN nodes for others.

I'm reporting this several hours after hearing about the issue, but even if the CDN's cache has been resolved in the interim, it would be good to know if we can avoid CDN cache misses like this if possible. /cc @skearns


(Sam Saffron) #2

I was just investigating:

https://discuss.elastic.co/uploads/default/original/2X/b/b0b0ab96024fe47abc5e6adac535ce3259f0997b.png

which is

https://discourse-cdn-sjc1.com/elastic/uploads/default/original/2X/b/b0b0ab96024fe47abc5e6adac535ce3259f0997b.png

it appears like the origin is missing. investigating this internally and will let you know what is up.

The category I am seeing the broken image on is https://discuss.elastic.co/c/x-pack , can not see an image when I edit the category as well.


(Alan Tan) #3

This is due to a bug we introduced sometime ago, the value of the upload path was stored incorrectly in the database which led to the image being destroyed by our clean up orphan uploads job. I've fixed the URLs on elastic search so the category logos will stop disappearing mysteriously.


(Tyler Langlois) #4

Many thanks @tgxworld!


(Nick) #5

Hey, I just noticed we seem still have this issue.


(Steve Kearns) #6

This is still occurring. Any updates? This is a long time for us to be serving 404 images to thousands of users.


(Alan Tan) #7

Hi @skearns,

We just deployed your instance for the proper fix. Apologies for the delay in my reply.


(Steve Kearns) #8

Hi Alan,

Thanks for your quick reply. I'm afraid it is now worse - now, instead of a 404 for the missing images, the whole image placeholder has been removed from a number of the category headers.

I'm concerned that this is so problematic for such a visible feature. Should we be concerned that other images within our discuss forum are disappearing without notice as well?

Please let us know how we can resolve this quickly.

Thanks,
Steve


(Alan Tan) #9

Hi Steve,

There were a couple of regressions with how we stored category logos and background images in the last couple of months. As a result, some category logos were incorrectly cleaned up by the clean up orphan uploads job. As we have a backup of all uploads, I've restored the images on to elastic.

Let me know if you still see anything missing :slight_smile:


(Steve Kearns) #10

Looks good now, thanks Alan!

Can we get some sort of automated testing in place to prevent regressions like this in the future?

Thanks
Steve


(Alan Tan) #11

Yup this has been fixed in

We were storing the upload URL on the model instead of properly creating an association previously.


(system) #12