Last modified: 2014-11-11 05:10:30 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T74898, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 72898 - Nightly builds should be a shallow git clone
Nightly builds should be a shallow git clone
Status: VERIFIED FIXED
Product: Pywikibot
Classification: Unclassified
General (Other open bugs)
core-(2.0)
All All
: High enhancement
: ---
Assigned To: Amir Ladsgroup
:
Depends on:
Blocks: pwb20
  Show dependency treegraph
 
Reported: 2014-11-03 12:37 UTC by John Mark Vandenberg
Modified: 2014-11-11 05:10 UTC (History)
3 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description John Mark Vandenberg 2014-11-03 12:37:33 UTC
The nightly builds appear to include a complete git clone.  There have been complaints about the time it takes to unzip the file.  It is also unfriendly to people not sitting on a high speed data unlimited connection.
Comment 1 Amir Ladsgroup 2014-11-03 13:34:37 UTC
I set up the nightly dumps and I'll fix it :)
Comment 2 Amir Ladsgroup 2014-11-03 21:07:59 UTC
I made some improvements, Is it still valid?
Comment 3 John Mark Vandenberg 2014-11-09 20:09:59 UTC
http://tools.wmflabs.org/pywikibot/core.tar.gz - 88M

http://tools.wmflabs.org/pywikibot/compat.tar.gz - 29M

It would be nice if http://tools.wmflabs.org/pywikibot/ showed the size of the download after each link.

Also there are broken links on http://tools.wmflabs.org/pywikibot/
Comment 4 Amir Ladsgroup 2014-11-10 17:24:46 UTC
Even with shallow clone loads of garbage (GARBAGE! I'm angry) were being downloaded (for instance 26M for compat, 14M for i18n).

At first I changed git pull to removing it completely and re-cloning it again. some improvements were made but not much.
I used git gc to reduce git objects but it didn't work out very well, I used more aggressive methods (like --prune=all and another git gc for i18n submodule) and now core is less than 10M and compat is 20M. The only thing that worries me it's that --prune=all maybe causes breakage of pwb.py version (or version.py in compat). I will check and tell you soon.

I also fixed broken links

About showing file sizes since it's plain html, It's not possible (rewriting that page in php won't be hard but another bug would be a better place for it)
Comment 5 John Mark Vandenberg 2014-11-11 00:00:10 UTC
http://tools.wmflabs.org/pywikibot/core.tar.gz - 8.5M - very nice!!  As mentioned, need to test version functionality.
Comment 6 Amir Ladsgroup 2014-11-11 04:44:12 UTC
Output:
amir@amir-Inspiron-N5110:~/Downloads/core$ python pwb.py version
Pywikibot: [https] r-pywikibot-core.git (47ba777, g3, 2014/11/10, 00:50:49, ok)
Release version: 2.0b2
Even though the the g number is nonsense but the date and first number is completely okay, output of git log is also okay.
Comment 7 John Mark Vandenberg 2014-11-11 04:52:26 UTC
and -verbose ?
Comment 8 Amir Ladsgroup 2014-11-11 05:09:24 UTC
version.py is the same with or without verbose (obviously):

A simple script with verbose looks okay to me:
http://paste.ubuntu.com/8933845/
Comment 9 John Mark Vandenberg 2014-11-11 05:10:30 UTC
Very nice!

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links