Last modified: 2014-03-31 15:40:25 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T64470, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 62470 - Migrate /data/project from math project into eqiad
Migrate /data/project from math project into eqiad
Status: RESOLVED FIXED
Product: Wikimedia Labs
Classification: Unclassified
Infrastructure (Other open bugs)
unspecified
All All
: Unprioritized minor
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks: 62042
  Show dependency treegraph
 
Reported: 2014-03-10 06:26 UTC by physikerwelt
Modified: 2014-03-31 15:40 UTC (History)
6 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description physikerwelt 2014-03-10 06:26:04 UTC
Hi,

could you please migrate or create a new folder for math in the eqiad project.
Otherwise new instances in eqiad can not be created. I deleted all custom puppet roles... but that did not help.
Thanks.
The puppet error is:
notice: /Stage[first]/Apt::Update/Exec[/usr/bin/apt-get update]/returns: executed successfully
notice: /Stage[main]/Role::Labs::Instance/Mount[/home]/ensure: ensure changed 'unmounted' to 'mounted'
err: /Stage[main]/Role::Labs::Instance/Mount[/home]: Could not evaluate: Execution of '/bin/mount -o rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc /home' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/math/home failed, reason given by server:
  No such file or directory

notice: /Stage[main]/Base::Puppet/Exec[puppet snmp trap]/returns: executed successfully
notice: instanceproject: math
notice: /Stage[main]/Base::Puppet/Notify[instanceproject: math]/message: defined 'message' as 'instanceproject: math'
notice: /Stage[main]/Role::Labs::Instance/Mount[/data/project]/ensure: ensure changed 'unmounted' to 'mounted'
err: /Stage[main]/Role::Labs::Instance/Mount[/data/project]: Could not evaluate: Execution of '/bin/mount -o rw,vers=4,bg,hard,intr,sec=sys,proto=tcp,port=0,noatime,nofsc /data/project' returned 32: mount.nfs: mounting labstore.svc.eqiad.wmnet:/project/math/project failed, reason given by server:
  No such file or directory
Comment 1 Andrew Bogott 2014-03-10 06:48:28 UTC
Ah, this is because shared volumes (both home and project) were turned off in the project settings.  We're going to have shared volumes for everyone in eqiad but that setting is still observed by some of the scripts.

I'm not sure what the proper general fix is, but in the meantime your project should be all set.
Comment 2 physikerwelt 2014-03-10 08:12:34 UTC
This seems to break compatibilty with labsvagrant

err: /Stage[main]/Labs_vagrant/User[vagrant]/ensure: change from absent to present failed: Could not create user vagrant: Execution of '/usr/sbin/useradd -m vagrant' returned 12: useradd: cannot create directory /home/vagrant
Comment 3 physikerwelt 2014-03-10 08:16:21 UTC
My feeling is that this is more a bug in the labsvagrant role.
Comment 4 Andrew Bogott 2014-03-10 09:26:33 UTC
If the labsvagrant class is broken then this should maybe have its own bug, or needs to be taken up with the people who wrote that class (yuvi and/or matanya).  This doesn't look like a failure of eqiad, although it may have to do with the way the nfs /home share is set up.
Comment 5 Gerrit Notification Bot 2014-03-11 06:33:31 UTC
Change 118053 had a related patch set uploaded by Physikerwelt:
Change home directory of vagrant user

https://gerrit.wikimedia.org/r/118053
Comment 6 Bryan Davis 2014-03-11 22:28:44 UTC
(In reply to physikerwelt from comment #2)
> err: /Stage[main]/Labs_vagrant/User[vagrant]/ensure: change from absent to
> present failed: Could not create user vagrant: Execution of
> '/usr/sbin/useradd -m vagrant' returned 12: useradd: cannot create directory
> /home/vagrant

My best guess is that this was caused by the intermittent problem of the NFS server making shared read-only due to a race in application of ACLs. Coren knows about this problem and is trying to track down it's root cause. It seems to be cache related and will typically clear after a reboot or a 10-15 minute wait.

There was a related problem that the NFSv4 server in eqiad is enforcing idmapd identities and the NFS server didn't know about the locally created vagrant user. Coren has fixed this by adding a vagrant user to the eqiad LDAP.

You may have to create the /home/vagrant directory manually at this point. This may be something that needs to be fixed in the labs_vagrant role.
Comment 7 physikerwelt 2014-03-17 06:31:41 UTC
In fact I had to create the /home/vagrant directory manually and fix permissions. Thus, I do not see the bug as fixed.
Comment 8 Gerrit Notification Bot 2014-03-30 09:13:08 UTC
Change 118053 merged by Ori.livneh:
Change home directory of vagrant user

https://gerrit.wikimedia.org/r/118053

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links