Last modified: 2012-01-09 18:32:41 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 10574 - Export pages should allow to export all pages
Export pages should allow to export all pages
Status: RESOLVED FIXED
Product: MediaWiki
Classification: Unclassified
Export/Import (Other open bugs)
unspecified
All All
: Low enhancement with 3 votes (vote)
: ---
Assigned To: Nobody - You can work on this!
: patch, patch-need-review
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-07-13 21:54 UTC by Thorsten Staerk
Modified: 2012-01-09 18:32 UTC (History)
11 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments
: exports all pages (1.88 KB, patch)
2007-07-13 21:57 UTC, Thorsten Staerk
Details
adds a button "add all pages" like "add pages from category" (3.09 KB, patch)
2007-07-14 10:30 UTC, Thorsten Staerk
Details
option to export all pages (3.80 KB, patch)
2007-07-15 20:37 UTC, Thorsten Staerk
Details
use $exporter to export all pages on request (3.04 KB, patch)
2007-07-19 06:57 UTC, Thorsten Staerk
Details
allow exporting all pages if less than $wgExportMaxPages (5.17 KB, patch)
2007-07-19 11:12 UTC, Thorsten Staerk
Details
after chatting with robchurch (5.31 KB, patch)
2007-07-19 12:17 UTC, Thorsten Staerk
Details
Allow export of all pages (46 bytes, text/plain)
2012-01-03 17:22 UTC, Greg Sabino Mullane
Details
Allow export of all pages (7.49 KB, patch)
2012-01-03 17:27 UTC, Greg Sabino Mullane
Details

Description Thorsten Staerk 2007-07-13 21:54:23 UTC
For small wikis, "export pages" should allow to export all pages.
Comment 1 Robert Leverington 2007-07-13 21:55:58 UTC
How do you define "small wikis" - would this be a configuration setting or some hard value. I would guess some wikis would rather not allow someone to download all of their content - especially while still small and relatively unknown.
Comment 2 Thorsten Staerk 2007-07-13 21:57:04 UTC
Created attachment 3907 [details]
: exports all pages

This patch allows to export all pages if you enter : as page name. However, there is still no description that this is possible.
Comment 3 Aryeh Gregor (not reading bugmail, please e-mail directly) 2007-07-13 22:29:43 UTC
I think you left some debugging code at the bottom there.  Also, it would be vastly more efficient to just directly select all pages using $exporter->allPages(), rather than generating a list of all pages (possibly a megabyte or more in memory on reasonable-sized wikis) and then going through those one-by-one.

If this is made available it needs to be a configuration option, obviously.  The interface should be improved, made a separate button instead of a magic title.
Comment 4 Thorsten Staerk 2007-07-14 10:30:51 UTC
Created attachment 3911 [details]
adds a button "add all pages" like "add pages from category"

absolutely right that we should rather give a gui option instead of a magic ":" page-name.
Comment 5 Thorsten Staerk 2007-07-14 12:15:38 UTC
Find a demo about this on https://staerk.dyndns.org/wiki
Comment 6 Thorsten Staerk 2007-07-14 20:36:00 UTC
I could just successfully migrate my mediawiki 1.6.10 to a mediawiki 1.11.0 (phase3). For that, I exported all pages from my old wiki and imported them into my new wiki. So, exporting can help you during migrations. Please take my patch to mainstream mediawiki.
Comment 7 Thorsten Staerk 2007-07-15 20:37:06 UTC
Created attachment 3915 [details]
option to export all pages
Comment 8 Aryeh Gregor (not reading bugmail, please e-mail directly) 2007-07-15 21:38:29 UTC
First of all, as I said, you should use $exporter->allPages(), not generate a list of all of them and go through those one by one.

Second of all, as for the configuration, I'm thinking it would be best to have that be in the form of a maximum number of pages that can be exported via Special:Export.  "Export all" could then be shown if the number of pages on the wiki is less than the provided number.  I say this because I saw nowhere in the code to restrict this, so I experimentally attempted to export the current version of over 200,000 files (the entire contents of Category:Living people) from the English Wikipedia.  It failed after like ten minutes with an XML parse error of some kind.  Probably that isn't desirable?  If you get a dump of all page names, which is simple, you can easily try to dump the current version of every page on the English Wikipedia through Special:Export.
Comment 9 Thorsten Staerk 2007-07-19 06:57:37 UTC
Created attachment 3928 [details]
use $exporter to export all pages on request

Here's my new version of the patch, it now uses $exporter->allPages(). I tested it for every combination I could think of. If you want to set some limits, please give me a hint how I can find the number of pages on the wiki. Else, please commit!
Comment 10 Rob Church 2007-07-19 07:45:51 UTC
You seem to be duplicating a lot of the export code path; just add conditional checks to the actual export statements. Use the Xml class methods, not the deprecated wfSubmitButton() et al.

You can use SiteStats::pages() to get the number of pages without doing an expensive COUNT(*). I would suggest introducing $wgExportMaxPages, which defaults to false (unlimited), and checking against it.
Comment 11 Thorsten Staerk 2007-07-19 11:12:33 UTC
Created attachment 3932 [details]
allow exporting all pages if less than $wgExportMaxPages

implemented all of RobChurch's ideas. Thanks, Rob and Simmetrical.
Comment 12 Thorsten Staerk 2007-07-19 12:17:50 UTC
Created attachment 3933 [details]
after chatting with robchurch

This is also preventing the user from adding manually more than $wgExportMaxPages
Comment 13 Rob Church 2007-07-19 20:36:12 UTC
Working on incorporating this into some forthcoming improvements to Special:Export
Comment 14 Thorsten Staerk 2007-08-06 18:07:12 UTC
I really think this is a great addition - can you give me svn access so I can commit that ?
I have already an svn account at KDE - e.g. at http://websvn.kde.org/trunk/KDE/kdepim/ktimetracker/ you can see that I (tstaerk) only commit thoroughly-tested code :)
Comment 15 Andrew Garrett 2007-09-10 09:25:50 UTC
There's a (very) dodgy, unlocalised patch that I wrote one bored night for exporting a whole category up at http://www.devanywhere.com/ViewPub.php?id=22 if anybody's vaguely interested in cleaning it up and using it.
Comment 16 Casey Brown 2007-09-10 20:08:50 UTC
(In reply to comment #14)
> I really think this is a great addition - can you give me svn access so I can
> commit that ?
> I have already an svn account at KDE - e.g. at
> http://websvn.kde.org/trunk/KDE/kdepim/ktimetracker/ you can see that I
> (tstaerk) only commit thoroughly-tested code :)
> 

You add an attachment, it is not necessary to have SVN access for a few bugs.  However, if you feel you need it, contact brion at wikimedia.org
Comment 17 Thorsten Staerk 2007-09-10 20:43:33 UTC
I do not feel I need svn access, I just would like to see my patch (attachment 6 [details]) to be committed.
Comment 18 Thorsten Staerk 2007-12-15 11:51:34 UTC
Anything new ?
Comment 19 Thorsten Staerk 2007-12-15 18:06:17 UTC
Rob, thanks for working on mediawiki, you brought a lot of innovation to it! Sad that you left
 
Thorsten
Comment 20 Siebrand Mazeland 2008-08-18 18:47:36 UTC
Mass compoment change: <some> -> Export/Import
Comment 21 Vitaliy Filippov 2010-03-18 14:23:12 UTC
See also Bug 22881 - Greatly improved Export and Import for 1.14.1 (with support for advanced page selection, exporting and importing file uploads, and detection of "conflicts" during import). There's a patch written by me which is related to or fixes your issue.
Comment 22 p858snake 2011-04-30 00:10:02 UTC
*Bulk BZ Change: +Patch to open bugs with patches attached that are missing the keyword*
Comment 23 Sumana Harihareswara 2011-11-10 06:11:26 UTC
Thorsten, I'm sorry for the wait on this.  I'm adding the "need-review" keyword to ask developers to review your code and approach.
Comment 24 Greg Sabino Mullane 2012-01-03 17:21:10 UTC
I came up with my own 'export all' solution before seeing this bug, but this seems the right place to post it. Similar in spirit to the earlier patches, but has a simple checkbox to export everything. Uses the load balancer per the large-history option, and has a global (defaults to false) that prevents the checkbox from appearing unless set to true. This feature is really needed for smaller wikis, else everyone ends up doing ugly workarounds such as putting all pages into a dummy category, or dumping each namespace separately. Diff is against r107904
Comment 25 Greg Sabino Mullane 2012-01-03 17:22:27 UTC
Created attachment 9791 [details]
Allow export of all pages
Comment 26 Greg Sabino Mullane 2012-01-03 17:27:12 UTC
Created attachment 9792 [details]
Allow export of all pages

Disregard previous URL, better to have the patch
Comment 27 Mark A. Hershberger 2012-01-03 19:47:55 UTC
(In reply to comment #24)
> This feature is really needed for smaller wikis, else everyone ends up doing
> ugly workarounds such as putting all pages into a dummy category, or dumping
> each namespace separately.

Would you mind applying this (referencing this bug, natch) so that we can look at it in CR?
Comment 28 Greg Sabino Mullane 2012-01-03 21:09:04 UTC
Applied to r107939
Comment 29 Greg Sabino Mullane 2012-01-05 17:10:30 UTC
Also r108151
Comment 30 Mark A. Hershberger 2012-01-09 18:32:41 UTC
closing this since the fixes are made.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links