Last modified: 2013-12-07 09:32:07 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 53227 - Stale non-slow maintenance Special pages on Wikimedia wikis: bi-weekly updateSpecialPages.php no longer successfully running
Stale non-slow maintenance Special pages on Wikimedia wikis: bi-weekly update...
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
General/Unknown (Other open bugs)
wmf-deployment
All All
: Highest major with 7 votes (vote)
: ---
Assigned To: Sam Reed (reedy)
:
: 45007 55483 55785 55910 56364 (view as bug list)
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2013-08-22 17:44 UTC by Umherirrender
Modified: 2013-12-07 09:32 UTC (History)
27 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
Log of manual run (1.54 MB, text/plain)
2013-08-26 17:55 UTC, Sam Reed (reedy)
Details

Description Umherirrender 2013-08-22 17:44:46 UTC
Last update of cached special pages on dewiki is 13.08. there are runs missing from 16.08., 19.08. and today (22.08.)

Please have a look, why the cron job or the script does not work. Needs (more) monitoring. Thanks.

"The following data is cached, and was last updated 14:33, 13 August 2013."

https://de.wikipedia.org/wiki/Spezial:Defekte_Weiterleitungen?uselang=en
https://de.wikipedia.org/wiki/Spezial:Doppelte_Weiterleitungen?uselang=en
Comment 1 Umherirrender 2013-08-22 18:08:13 UTC
Have a look at bug 45007 comment 4
Comment 2 Sam Reed (reedy) 2013-08-23 13:56:23 UTC
reedy@terbium:/home/mwdeploy/updateSpecialPages$ ls -al s5@15-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 455 Aug 15 01:02 s5@15-DeadendPages.log
reedy@terbium:/home/mwdeploy/updateSpecialPages$ cat s5@15-DeadendPages.log
-------------------------------------
dewiki
-------------------------------------
Statistics                     completed in 10.83s
UnreviewedPages                completed in 9.71s
ValidationStatistics           completed in 7.51s
-------------------------------------

-------------------------------------
wikidatawiki
-------------------------------------
Statistics                     2m completed in 13.16s
-------------------------------------
Comment 3 Sam Reed (reedy) 2013-08-23 14:10:09 UTC
reedy@hume:/home/wikipedia/log/norotate$ flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /home/wikipedia/logs/norotate/updateSpecialPages.log 2>&1
reedy@hume:/home/wikipedia/log/norotate$
Comment 4 Gerrit Notification Bot 2013-08-23 14:12:26 UTC
Change 80560 had a related patch set uploaded by Reedy:
Maintenance scripts should be run as Apache

https://gerrit.wikimedia.org/r/80560
Comment 5 Gerrit Notification Bot 2013-08-23 14:13:41 UTC
Change 80560 abandoned by Reedy:
Maintenance scripts should be run as Apache

Reason:
user => "apache",

https://gerrit.wikimedia.org/r/80560
Comment 6 Sam Reed (reedy) 2013-08-23 14:28:07 UTC
Monitor the current manual run via http://noc.wikimedia.org/~reedy/updateSpecialPages.log
Comment 7 Dennis C. During 2013-08-23 18:46:42 UTC
So, it's going to be a while before we get to enwikt?
Comment 8 Sam Reed (reedy) 2013-08-24 01:02:10 UTC
And a lot longer to zhwiki
Comment 9 This, that and the other (TTO) 2013-08-26 07:12:16 UTC
No patch to review, so changing to NEW.
Comment 10 Sam Reed (reedy) 2013-08-26 17:55:12 UTC
Created attachment 13170 [details]
Log of manual run

It has apparently run to completion on my manual run
Comment 11 Malafaya 2013-09-09 22:20:24 UTC
Apparently, no update by cron ran since 13/08. The last 24/08 run was a manual one (see comments above).
Comment 12 Nemo 2013-09-27 07:36:37 UTC
*** Bug 45007 has been marked as a duplicate of this bug. ***
Comment 13 Andre Klapper 2013-09-27 09:54:33 UTC
Reedy: Do you know what's the plan for this issue?

Also brought up at https://en.wikipedia.org/wiki/Wikipedia:Village_pump_%28technical%29#Cached_special_pages_not_being_updated
Comment 14 Umherirrender 2013-10-07 18:24:44 UTC
Now the last run is a month ago (10.09)

Please have a look at the script/cronjob. Thanks.
Comment 15 Malafaya 2013-10-08 18:11:04 UTC
Still not running successfully with cron, it seems.
I'm already afraid of asking for yet another manual run... :)
Comment 16 Sam Reed (reedy) 2013-10-08 21:11:42 UTC
*** Bug 55483 has been marked as a duplicate of this bug. ***
Comment 17 Malafaya 2013-10-14 14:48:37 UTC
What's the current status of this issue? What is being done currently?
Comment 18 Bartosz Dziewoński 2013-10-16 12:03:19 UTC
*** Bug 55785 has been marked as a duplicate of this bug. ***
Comment 19 Bartosz Dziewoński 2013-10-16 12:20:20 UTC
*** Bug 55785 has been marked as a duplicate of this bug. ***
Comment 20 Gerrit Notification Bot 2013-10-16 13:36:39 UTC
Change 90117 had a related patch set uploaded by Nemo bis:
Simplify misc::maintenance::update_special_pages a bit

https://gerrit.wikimedia.org/r/90117
Comment 21 Umherirrender 2013-10-19 12:40:28 UTC
*** Bug 55910 has been marked as a duplicate of this bug. ***
Comment 22 Bartosz Dziewoński 2013-10-19 13:53:43 UTC
Is anything being done about this? Reedy?
Comment 23 MZMcBride 2013-10-19 13:54:53 UTC
(In reply to comment #22)
> Is anything being done about this? Reedy?

I just had a similar question. Check out comment 20? :-)
Comment 24 Bartosz Dziewoński 2013-10-19 13:57:07 UTC
I can't tell what that patch is supposed to do, but it looks like some general cleanup, not a fix to this.
Comment 25 MZMcBride 2013-10-19 14:00:44 UTC
Are there relevant logs that someone can look at? It's unclear to me what the problem is.
Comment 26 Superyetkin 2013-10-20 08:13:20 UTC
(In reply to comment #24)
> I can't tell what that patch is supposed to do, but it looks like some
> general
> cleanup, not a fix to this.

I agree. It would be helpful to see where the script stops running.
Comment 27 Nemo 2013-10-20 08:58:58 UTC
(In reply to comment #26)
> I agree. It would be helpful to see where the script stops running.

Last time I checked, even the aa/ab wikis didn't have any update since September, so it fails in very early stages.
Comment 28 Umherirrender 2013-10-22 16:20:17 UTC
It sounds like a problem with the cron job, because the manuell run completed succesfully, but a automatic run does not start.

Mabye a operator can have a look here.

For the moment a new manually run where nice. Thanks for that.
Comment 29 billinghurst 2013-10-26 08:59:46 UTC
For enWS it is reported to have started failing at 10 September, which aligns with something in 1.22wmf16 according to
https://www.mediawiki.org/wiki/MediaWiki_1.22/Roadmap
Comment 30 This, that and the other (TTO) 2013-10-30 08:41:51 UTC
*** Bug 56364 has been marked as a duplicate of this bug. ***
Comment 31 Umherirrender 2013-10-31 17:39:53 UTC
Please run updateSpecialPages.php manually on all wikis or at least dewiki. Thanks!
Comment 32 Yellowcard 2013-11-01 12:27:33 UTC
Can someone please start to fix this? Thanks.
Comment 33 Hexachloroiridic Acid 2013-11-01 15:14:28 UTC
Special-page-updating is still non-functional. 
I cannot tell, if this is a worldwide issue or just affecting dewiki, however dewiki is missing special page updates since September 2013.

Please take a look at the issue.
Thanks!
Comment 34 Andre Klapper 2013-11-01 15:20:29 UTC
Sorry that this takes longer - higher priority issues got into the way for the last days. 
I assume that Reedy will find time next week again to look into this again.
Comment 35 Umherirrender 2013-11-05 18:25:18 UTC
Please run updateSpecialPages.php manually on all wikis. Thanks!
Comment 36 MZMcBride 2013-11-05 19:25:11 UTC
Greg or Sam: my question in comment 25 is still unanswered.
Comment 37 Sam Reed (reedy) 2013-11-05 23:47:49 UTC
(In reply to comment #36)
> Greg or Sam: my question in comment 25 is still unanswered.

What a shame.

(In reply to comment #25)
> Are there relevant logs that someone can look at? It's unclear to me what the
> problem is.

Relevant logs for what exactly?

grep is such a useful tool

eedy@ubuntu64-web-esxi:~/git/operations/puppet$ grep -r updateSpecialPages *
files/misc/scripts/update-special-pages:        php MWScript.php updateSpecialPages.php $db
manifests/misc/maintenance.pp:                  command => "flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /var/log/updateSpecialPages.log 2>&1",
manifests/misc/maintenance.pp:        # Include this to add cron jobs calling updateSpecialPages.php on all clusters.
manifests/misc/maintenance.pp:        file { '/home/mwdeploy/updateSpecialPages':
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Ancientpages > /home/mwdeploy/updateSpecialPages/${name}-AncientPages.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Deadendpages > /home/mwdeploy/updateSpecialPages/${name}-DeadendPages.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Mostlinked > /home/mwdeploy/updateSpecialPages/${name}-MostLinked.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Mostrevisions > /home/mwdeploy/updateSpecialPages/${name}-MostRevisions.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Wantedpages > /home/mwdeploy/updateSpecialPages/${name}-WantedPages.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php ${cluster}.dblist --override --only=Fewestrevisions > /home/mwdeploy/updateSpecialPages/${name}-FewestRevisions.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Lonelypages > /home/mwdeploy/updateSpecialPages/${name}-LonelyPages.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Mostcategories > /home/mwdeploy/updateSpecialPages/${name}-MostCategories.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Mostlinkedcategories > /home/mwdeploy/updateSpecialPages/${name}-MostLinkedCategories.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Mostlinkedtemplates > /home/mwdeploy/updateSpecialPages/${name}-MostLinkedTemplates.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Uncategorizedcategories > /home/mwdeploy/updateSpecialPages/${name}-UncategorizedCategories.log 2>&1",
manifests/misc/maintenance.pp:                        command => "/usr/local/bin/mwscriptwikiset updateSpecialPages.php s1.dblist --override --only=Wantedtemplates > /home/mwdeploy/updateSpecialPages/${name}-WantedTemplates.log 2>&1",
reedy@ubuntu64-web-esxi:~/git/operations/puppet$
Comment 38 Sam Reed (reedy) 2013-11-06 01:06:33 UTC
reedy@terbium:/home/mwdeploy/updateSpecialPages$ ls -al
total 1072
drwxrwxr-x 2 mwdeploy mwdeploy   4096 Oct 21 01:00 .
drwxrwxr-x 4 mwdeploy mwdeploy   4096 Jul 30 21:38 ..
-rw-rw-r-- 1 mwdeploy mwdeploy    183 Oct 11 05:02 s1@11-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    183 Oct 11 04:56 s1@11-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    185 Oct 11 22:23 s1@11-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    185 Oct 12 00:59 s1@11-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy    184 Oct 11 10:54 s1@11-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    185 Oct 12 04:54 s1@11-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy   3075 Oct 12 04:17 s2@12-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy   3050 Oct 12 03:19 s2@12-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy   3084 Oct 12 06:43 s2@12-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy   3090 Oct 12 11:25 s2@12-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy   3082 Oct 12 06:37 s2@12-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy   3082 Oct 12 06:29 s2@12-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151521 Oct 13 03:27 s3@13-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 150670 Oct 13 02:53 s3@13-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151457 Oct 13 03:52 s3@13-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151677 Oct 13 08:28 s3@13-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151455 Oct 13 03:54 s3@13-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy 151341 Oct 13 05:36 s3@13-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    185 Oct 14 01:06 s4@14-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    182 Oct 14 01:00 s4@14-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    186 Oct 14 01:11 s4@14-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    189 Oct 14 05:25 s4@14-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy    185 Oct 14 01:08 s4@14-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    185 Oct 14 01:53 s4@14-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    369 Oct 15 02:46 s5@15-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    360 Oct 15 01:28 s5@15-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    370 Oct 15 05:12 s5@15-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    372 Oct 15 04:15 s5@15-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy    372 Oct 15 05:11 s5@15-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    369 Oct 15 02:21 s5@15-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    543 Oct 16 02:00 s6@16-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    537 Oct 16 02:29 s6@16-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    544 Oct 16 03:50 s6@16-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    551 Oct 16 06:55 s6@16-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy    546 Oct 16 03:47 s6@16-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy    546 Oct 16 03:57 s6@16-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy   2170 Oct 17 02:22 s7@17-AncientPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy   2149 Oct 17 02:07 s7@17-DeadendPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy   2172 Oct 17 03:40 s7@17-FewestRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy   2181 Oct 17 08:17 s7@17-MostLinked.log
-rw-rw-r-- 1 mwdeploy mwdeploy   2173 Oct 17 03:40 s7@17-MostRevisions.log
-rw-rw-r-- 1 mwdeploy mwdeploy   2179 Oct 17 05:09 s7@17-WantedPages.log
-rw-rw-r-- 1 mwdeploy mwdeploy    184 Oct 21 09:22 updatequerypages-enwiki-only-MostLinkedTemplates.log
reedy@terbium:/home/mwdeploy/updateSpecialPages$
Comment 39 Sam Reed (reedy) 2013-11-06 01:15:56 UTC
(In reply to comment #35)
> Please run updateSpecialPages.php manually on all wikis. Thanks!

Comments like that really aren't helping anyone.

From hume
-rw-rw-r-- 1 reedy     wikidev    1614045 Sep 11 02:35 updateSpecialPages.log
-rw-rw-r-- 1 apache    wikidev     951081 Feb  4  2013 updateSpecialPages-small.log











Does anyone actually know what isn't working? "It doesn't work" or "They're not being run" is no help to anyone


class misc::maintenance::update_special_pages( $enabled = false ) {
        cron {
                update_special_pages:
                        command => "flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /var/log/updateSpecialPages.log 2>&1",
                        user => "apache",
                        monthday => "*/3",
                        hour => 5,
                        minute => 0,
                        ensure => $enabled ?{
                                true => present,
                                false => absent,
                                default => absent
                        };
                update_special_pages_small:
                        ensure => absent;
        }

        file {
                "/usr/local/bin/update-special-pages":
                        source => "puppet:///files/misc/scripts/update-special-pages",
                        owner => apache,
                        group => wikidev,
                        mode => 0755,
                        ensure => present;
                "/usr/local/bin/update-special-pages-small":
                        ensure => absent;
        }
}

Enabled on terbium according to site.pp

reedy@terbium:~$ ls -al /var/log/updateSpecialPages.log
ls: cannot access /var/log/updateSpecialPages.log: No such file or directory
reedy@terbium:~$
reedy@terbium:~$ ls -al /var/lock/update-special-pages
-rw-rw-r-- 1 apache apache 0 Apr 16  2013 /var/lock/update-special-pages
reedy@terbium:~$ ls -al /usr/local/bin/update-special-pages
-rwxr-xr-x 1 apache wikidev 196 Mar 26  2013 /usr/local/bin/update-special-pages
reedy@terbium:~$ cat /usr/local/bin/update-special-pages
#!/bin/bash

. /usr/local/lib/mw-deployment-vars.sh

cd "$MW_COMMON/multiversion"
for db in `<"$MW_DBLISTS/all.dblist"`; do
        echo $db
        php MWScript.php updateSpecialPages.php $db
        echo
        echo
done
reedy@terbium:~$

The job is in the crontab of the apache user

# Puppet Name: update_special_pages
0 5 */3 * * flock -n /var/lock/update-special-pages /usr/local/bin/update-special-pages > /var/log/updateSpecialPages.log 2>&1




The next place would be to check the cron logs, which I cannot do as I cannot read /var/log/syslog
Comment 40 Daniel Zahn 2013-11-06 01:47:19 UTC
- confirmed this is on terbium in the cronjob of apache
- but that logfile doesn't exist, so didn't look like it runs
- tried with "sudo -u apache .." manually, worked

- debugged a little and touched the logfile manually, let apache user write to it, changed cronjob to "in a minute" (i commented the line actually executing mwscript) and then it worked, as opposed to before, when i could see the cron job starting in syslog but no log appearing

so i say next time it should run now that i changed the logfile permissions, and the root cause would be having to puppetize this file and it's permissions
Comment 41 Nemo 2013-11-06 09:16:49 UTC
(In reply to comment #40)
> so i say next time it should run now that i changed the logfile permissions,
> and the root cause would be having to puppetize this file and it's
> permissions

How about /var/lock/update-special-pages , did you change permissions for that too? It's not clear to me why we even need flock (see patch above).
Comment 42 Malafaya 2013-11-06 13:50:49 UTC
> 
> Does anyone actually know what isn't working? "It doesn't work" or "They're
> not
> being run" is no help to anyone
> 

Check https://pt.wiktionary.org/wiki/Especial:Categorias_pedidas?uselang=en . Last updated "10 September 2013". That's the farthest I can check.
Comment 43 This, that and the other (TTO) 2013-11-11 11:32:17 UTC
Special pages ran on 8 November and again today.
Comment 44 billinghurst 2013-11-11 11:46:00 UTC
Confirmed at enWS. Thanks to those involved.
Comment 45 Dennis C. During 2013-11-11 13:47:42 UTC
Thanks from en.wikt for getting this done.

Were these two runs special, requiring intervention not part of routine operations?
Comment 46 Nemo 2013-11-11 14:17:04 UTC
(In reply to comment #45)
> Were these two runs special, requiring intervention not part of routine
> operations?

No, it was the automatic cronjob. As said above, Daniel fixed the permissions of some things.
Comment 47 Gerrit Notification Bot 2013-12-07 09:32:07 UTC
Change 90117 abandoned by Nemo bis:
Simplify misc::maintenance::update_special_pages a bit

Reason:
not worth it

https://gerrit.wikimedia.org/r/90117

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links