Last modified: 2013-04-11 22:45:42 UTC
There are several recent threads about thumbnail cache purging issues. Some currently still miss more information and this could turn out to be more than one problem, but listing them here: * https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Problems_with_images * https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#New_vector_image_won.27t_update * https://en.wikipedia.org/wiki/User_talk:Nowyouseeme#Question about https://en.wikipedia.org/wiki/File:US_Senators_Supporting_SSM.svg This one has nice testcase: * https://en.wikipedia.org/wiki/Wikipedia:HD#Purge_image_thumbnail I can confirm that https://upload.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png sometimes shows the old outdated version, but most of the time shows the updated version (I'm in Europe). I've went to https://upload.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png?RandomNumberHere123456789 and then went to https://en.wikipedia.org/wiki/File:NRHP_Illustrated_Counties.svg?action=purge but the problem still persists. $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png' --2013-04-07 01:43:03-- http://upload-lb.eqiad.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png Resolving upload-lb.eqiad.wikimedia.org... 208.80.154.235, 2620:0:861:ed1a::b Connecting to upload-lb.eqiad.wikimedia.org|208.80.154.235|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK X-Object-Meta-Sha1base36: dql1a3r1xtnnop4b5i2xbnbnl2ko53y Content-Disposition: inline;filename*=UTF-8''NRHP_Illustrated_Counties.svg.png Last-Modified: Wed, 03 Apr 2013 06:29:14 GMT Etag: ef9acb99c896a9e9ec19047ee399d96a X-Timestamp: 1364970554.37424 Content-Type: image/png X-Varnish: 2378273142 2051365130, 4237763979 Via: 1.1 varnish, 1.1 varnish Content-Length: 127301 Accept-Ranges: bytes Date: Sat, 06 Apr 2013 23:43:03 GMT Age: 215326 Connection: keep-alive X-Cache: cp1028 hit (536), cp1034 frontend miss (0) Access-Control-Allow-Origin: * Length: 127301 (124K) [image/png] Saving to: “400px-NRHP_Illustrated_Counties.svg.png.1” $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png' --2013-04-07 01:43:18-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK X-Object-Meta-Sha1base36: dql1a3r1xtnnop4b5i2xbnbnl2ko53y Content-Disposition: inline;filename*=UTF-8''NRHP_Illustrated_Counties.svg.png Last-Modified: Wed, 03 Apr 2013 06:29:14 GMT Etag: ef9acb99c896a9e9ec19047ee399d96a X-Timestamp: 1364970554.37424 Content-Type: image/png X-Varnish: 2377693018 2051365130, 707352624 707109699, 2426462443 2426369331 Via: 1.1 varnish, 1.1 varnish, 1.1 varnish Content-Length: 127301 Accept-Ranges: bytes Date: Sat, 06 Apr 2013 23:43:18 GMT Age: 215340 Connection: keep-alive X-Cache: cp1028 hit (534), cp3006 hit (3), cp3004 frontend hit (1) Access-Control-Allow-Origin: * Length: 127301 (124K) [image/png] Saving to: “400px-NRHP_Illustrated_Counties.svg.png.2”
*** Bug 46931 has been marked as a duplicate of this bug. ***
So maybe varnishhtcpd is broken again? (everywhere)? When I get a squid responding from esams things seem purged fine (This was directly after purging the image): bawolff@Bawolff-L:~$ wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png' --2013-04-09 08:47:38-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK Content-Type: image/png Content-Length: 133559 Content-Disposition: inline;filename*=UTF-8''NRHP_Illustrated_Counties.svg.png Last-Modified: Tue, 09 Apr 2013 11:45:59 GMT Access-Control-Allow-Origin: * Date: Tue, 09 Apr 2013 11:45:59 GMT X-Cache: MISS from sq86.wikimedia.org X-Cache-Lookup: MISS from sq86.wikimedia.org:3128 X-Cache: MISS from amssq55.esams.wikimedia.org X-Cache-Lookup: MISS from amssq55.esams.wikimedia.org:3128 X-Cache: MISS from amssq54.esams.wikimedia.org X-Cache-Lookup: MISS from amssq54.esams.wikimedia.org:80 Connection: keep-alive Length: 133559 (130K) [image/png] Saving to: `400px-NRHP_Illustrated_Counties.svg.png.7' 100%[======================================>] 133,559 3.19K/s in 25s 2013-04-09 08:48:06 (5.28 KB/s) - `400px-NRHP_Illustrated_Counties.svg.png.7' saved [133559/133559] --------------- Any response from a varnish (in either data center) gives me an age of 431614, even after purging the file on wiki.
Is there a good example with two different lengths/ages ? those looked like the same picture. the multicast purging is not broken everywhere, fyi.
$:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png'strated_Countie--2013-04-10 00:25:50-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK X-Object-Meta-Sha1base36: 3p9gzgrh65yoccesw52bmjw2um7n9b8 Content-Disposition: inline;filename*=UTF-8''NRHP_Illustrated_Counties.svg.png X-Timestamp: 1365545851.31050 Accept-Ranges: bytes Content-Length: 133559 Access-Control-Allow-Origin: * Date: Tue, 09 Apr 2013 22:23:03 GMT Content-Type: image/png Last-Modified: Tue, 09 Apr 2013 22:17:31 GMT ETag: e907d3765048cfc9ad63748ed19229bb X-Cache: HIT from sq86.wikimedia.org X-Cache-Lookup: HIT from sq86.wikimedia.org:3128 Age: 167 X-Cache: HIT from amssq55.esams.wikimedia.org X-Cache-Lookup: HIT from amssq55.esams.wikimedia.org:3128 X-Cache: MISS from amssq60.esams.wikimedia.org X-Cache-Lookup: MISS from amssq60.esams.wikimedia.org:80 Connection: keep-alive Length: 133559 (130K) [image/png] Saving to: “400px-NRHP_Illustrated_Counties.svg.png” 100%[===================================================================================================>] 133,559 450K/s in 0.3s 2013-04-10 00:25:50 (450 KB/s) - “400px-NRHP_Illustrated_Counties.svg.png” saved [133559/133559] $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png' --2013-04-10 00:25:53-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/c/cb/NRHP_Illustrated_Counties.svg/400px-NRHP_Illustrated_Counties.svg.png Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK X-Object-Meta-Sha1base36: dql1a3r1xtnnop4b5i2xbnbnl2ko53y Content-Disposition: inline;filename*=UTF-8''NRHP_Illustrated_Counties.svg.png Last-Modified: Wed, 03 Apr 2013 06:29:14 GMT Etag: ef9acb99c896a9e9ec19047ee399d96a X-Timestamp: 1364970554.37424 Content-Type: image/png X-Varnish: 2744905806 2051365130, 1035441300 1035100289, 1212131155 Via: 1.1 varnish, 1.1 varnish, 1.1 varnish Content-Length: 127301 Accept-Ranges: bytes Date: Tue, 09 Apr 2013 22:25:53 GMT Age: 469896 Connection: keep-alive X-Cache: cp1028 hit (1064), cp3006 hit (7), cp3009 frontend miss (0) Access-Control-Allow-Origin: * Length: 127301 (124K) [image/png] Saving to: “400px-NRHP_Illustrated_Counties.svg.png.1” 100%[===================================================================================================>] 127,301 442K/s in 0.3s 2013-04-10 00:25:53 (442 KB/s) - “400px-NRHP_Illustrated_Counties.svg.png.1” saved [127301/127301]
cp1028 is misbehaving right now ... purging cache on that varnish and restarting varnishhtcpd seems to have fixed this - another example please?
And from dup bug 46931: $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/6/64/Codomain2.SVG' --2013-04-10 00:41:06-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/6/64/Codomain2.SVG Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK X-Object-Meta-Sha1base36: kzwmp46ues2ukmty1ytvuhpi5abska0 Last-Modified: Tue, 17 Jul 2012 02:21:23 GMT Etag: e0ab735d5cf72a80e1017cad46e26c45 X-Timestamp: 1342491683.95243 Content-Type: image/svg+xml X-Varnish: 1337403470 2081734694, 755800752 463732489, 1470008334 Via: 1.1 varnish, 1.1 varnish, 1.1 varnish Content-Length: 4155 Accept-Ranges: bytes Date: Tue, 09 Apr 2013 22:41:06 GMT Age: 2409675 Connection: keep-alive X-Cache: cp1033 hit (90), cp3005 hit (13), cp3010 frontend miss (0) Access-Control-Allow-Origin: * Length: 4155 (4.1K) [image/svg+xml] Saving to: “Codomain2.SVG” 100%[===================================================================================================>] 4,155 --.-K/s in 0.002s 2013-04-10 00:41:06 (2.04 MB/s) - “Codomain2.SVG” saved [4155/4155] $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/6/64/Codomain2.SVG' --2013-04-10 00:41:09-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/6/64/Codomain2.SVG Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK X-Object-Meta-Sha1base36: 79kmith7igibvfw3qspag91a5zrlqm7 Last-Modified: Fri, 05 Apr 2013 00:16:22 GMT ETag: 88d9ca8e221079a1277428313e9399ee X-Timestamp: 1365120982.99758 Accept-Ranges: bytes Content-Length: 10389 Content-Type: image/svg+xml Access-Control-Allow-Origin: * Date: Tue, 09 Apr 2013 00:54:27 GMT X-Cache: HIT from sq83.wikimedia.org X-Cache-Lookup: HIT from sq83.wikimedia.org:3128 Age: 78402 X-Cache: HIT from amssq52.esams.wikimedia.org X-Cache-Lookup: HIT from amssq52.esams.wikimedia.org:3128 X-Cache: MISS from knsq22.esams.wikimedia.org X-Cache-Lookup: MISS from knsq22.esams.wikimedia.org:80 Connection: keep-alive Length: 10389 (10K) [image/svg+xml] Saving to: “Codomain2.SVG.1” 100%[===================================================================================================>] 10,389 --.-K/s in 0.05s 2013-04-10 00:41:09 (222 KB/s) - “Codomain2.SVG.1” saved [10389/10389] $:andre\>
fixed with a restart of varnishhtcpd on cp1033
Darn, one more (from https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#New_vector_image_won.27t_update ): $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/en/thumb/2/2e/Duke_Energy_logo.svg/200px-Duke_Energy_logo.svg.png' --2013-04-10 00:57:42-- http://upload-lb.esams.wikimedia.org/wikipedia/en/thumb/2/2e/Duke_Energy_logo.svg/200px-Duke_Energy_logo.svg.png Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK X-Object-Meta-Sha1base36: ofbyhhqcou4td4xvqkfjh7en8qw0gl5 Content-Disposition: inline;filename*=UTF-8''Duke_Energy_logo.svg.png Last-Modified: Tue, 09 Apr 2013 12:50:13 GMT ETag: c6ebf076718f2b47900ee73e00a616d4 X-Timestamp: 1365511813.90989 Accept-Ranges: bytes Content-Length: 7072 Content-Type: image/png Access-Control-Allow-Origin: * Date: Tue, 09 Apr 2013 22:57:04 GMT X-Cache: MISS from sq51.wikimedia.org X-Cache-Lookup: MISS from sq51.wikimedia.org:3128 Age: 38 X-Cache: HIT from amssq48.esams.wikimedia.org X-Cache-Lookup: HIT from amssq48.esams.wikimedia.org:3128 X-Cache: MISS from amssq57.esams.wikimedia.org X-Cache-Lookup: MISS from amssq57.esams.wikimedia.org:80 Connection: keep-alive Length: 7072 (6.9K) [image/png] Saving to: “200px-Duke_Energy_logo.svg.png” 100%[===================================================================================================>] 7,072 --.-K/s in 0.001s 2013-04-10 00:57:42 (4.67 MB/s) - “200px-Duke_Energy_logo.svg.png” saved [7072/7072] $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/en/thumb/2/2e/Duke_Energy_logo.svg/200px-Duke_Energy_logo.svg.png' --2013-04-10 00:57:44-- http://upload-lb.esams.wikimedia.org/wikipedia/en/thumb/2/2e/Duke_Energy_logo.svg/200px-Duke_Energy_logo.svg.png Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Last-Modified: Thu, 26 Apr 2012 09:26:23 GMT Etag: ca5ea9d1aa26c7fa662829a7cd6370cc X-Timestamp: 1335432383.88755 Content-Type: image/png X-Varnish: 3355850179 1834259508, 1206612340 1206573200, 1215317841 Via: 1.1 varnish, 1.1 varnish, 1.1 varnish Content-Length: 4505 Accept-Ranges: bytes Date: Tue, 09 Apr 2013 22:57:44 GMT Age: 2084590 Connection: keep-alive X-Cache: cp1029 hit (3976), cp3007 hit (3), cp3009 frontend miss (0) Access-Control-Allow-Origin: * Length: 4505 (4.4K) [image/png] Saving to: “200px-Duke_Energy_logo.svg.png.1” 100%[===================================================================================================>] 4,505 --.-K/s in 0s 2013-04-10 00:57:44 (13.3 MB/s) - “200px-Duke_Energy_logo.svg.png.1” saved [4505/4505]
All aforementioned known issues should be fixed now, but might need https://en.wikipedia.org/wiki/WP:Purge . For the records, the middleterm improvement to identify such situations faster would be bug 43449. Closing as FIXED. Please open a new ticket for any new problems. Thanks!
It wasn't just thumbs, also originals (since April 5 at least). I've tested about ten reuploads in last hour on Commons and it seems fine indeed. Thanks! https://commons.wikimedia.org/w/index.php?title=Special:Log&limit=500&type=upload
Another one from https://commons.wikimedia.org/wiki/Commons:Village_pump#File:Boeing_X-45A_UCAV.jpg : $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/b/b1/Boeing_X-45A_UCAV.jpg/120px-Boeing_X-45A_UCAV.jpg' --2013-04-10 18:57:16-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/b/b1/Boeing_X-45A_UCAV.jpg/120px-Boeing_X-45A_UCAV.jpg Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.1 200 OK Last-Modified: Fri, 13 Apr 2012 16:31:55 GMT Etag: 961f99168bd957ef96ebf22a0778f735 X-Timestamp: 1334334715.36100 Content-Type: image/jpeg X-Varnish: 1829343736 612921789, 1274563771 1221166242, 1559320859 Via: 1.1 varnish, 1.1 varnish, 1.1 varnish Content-Length: 22448 Accept-Ranges: bytes Date: Wed, 10 Apr 2013 16:57:16 GMT Age: 1225264 Connection: keep-alive X-Cache: cp1027 hit (553), cp3009 hit (146), cp3005 frontend miss (0) Access-Control-Allow-Origin: * Length: 22448 (22K) [image/jpeg] Saving to: “120px-Boeing_X-45A_UCAV.jpg.3” 100%[===================================================================================================>] 22,448 --.-K/s in 0.05s 2013-04-10 18:57:17 (429 KB/s) - “120px-Boeing_X-45A_UCAV.jpg.3” saved [22448/22448] $:andre\> wget -S -U Malyacko --header 'host: upload.wikimedia.org' 'http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/b/b1/Boeing_X-45A_UCAV.jpg/120px-Boeing_X-45A_UCAV.jpg' --2013-04-10 18:57:18-- http://upload-lb.esams.wikimedia.org/wikipedia/commons/thumb/b/b1/Boeing_X-45A_UCAV.jpg/120px-Boeing_X-45A_UCAV.jpg Resolving upload-lb.esams.wikimedia.org... 91.198.174.234, 2620:0:862:ed1a::b Connecting to upload-lb.esams.wikimedia.org|91.198.174.234|:80... connected. HTTP request sent, awaiting response... HTTP/1.0 200 OK X-Object-Meta-Sha1base36: 4esak7p1lvd0l7niuali6zk0hf5k4s0 Content-Disposition: inline;filename*=UTF-8''Boeing_X-45A_UCAV.jpg X-Timestamp: 1365569436.27440 Accept-Ranges: bytes Content-Length: 1808 Access-Control-Allow-Origin: * Date: Wed, 10 Apr 2013 13:47:28 GMT Content-Type: image/jpeg Last-Modified: Wed, 10 Apr 2013 04:50:36 GMT ETag: c5e1761ba1968fb0c717601003fd7a98 X-Cache: HIT from sq85.wikimedia.org X-Cache-Lookup: HIT from sq85.wikimedia.org:3128 Age: 11390 X-Cache: HIT from amssq49.esams.wikimedia.org X-Cache-Lookup: HIT from amssq49.esams.wikimedia.org:3128 X-Cache: MISS from amssq55.esams.wikimedia.org X-Cache-Lookup: MISS from amssq55.esams.wikimedia.org:80 Connection: keep-alive Length: 1808 (1.8K) [image/jpeg] Saving to: “120px-Boeing_X-45A_UCAV.jpg.4” 100%[===================================================================================================>] 1,808 --.-K/s in 0s 2013-04-10 18:57:18 (5.71 MB/s) - “120px-Boeing_X-45A_UCAV.jpg.4” saved [1808/1808] $:andre\>
Comment 11 also fixed by Leslie, I'm going to comment on the Village Pump thread.
(In reply to comment #11) > Another one from > https://commons.wikimedia.org/wiki/Commons:Village_pump#File:Boeing_X- > 45A_UCAV.jpg That had been reuploaded before the varnishes were fixed, so it's not a good example. Purging worked. Also, cp1029 is affected by this server admin log entry of today and probably had some shakeup? 15:48 mark: Ran dist-upgrade (for varnish upgrade) on cp1021-1036 With changes like <https://gerrit.wikimedia.org/r/#/q/status:merged+project:operations/debs/varnish+branch:testing/3.0.3plus-rc1,n,z>
Looks great. Thanks everyone!