Last modified: 2013-03-26 11:25:13 UTC

Wikimedia Bugzilla is closed!

Wikimedia has migrated from Bugzilla to Phabricator. Bug reports should be created and updated in Wikimedia Phabricator instead. Please create an account in Phabricator and add your Bugzilla email address to it.
Wikimedia Bugzilla is read-only. If you try to edit or create any bug report in Bugzilla you will be shown an intentional error message.
In order to access the Phabricator task corresponding to a Bugzilla report, just remove "static-" from its URL.
You could still run searches in Bugzilla or access your list of votes but bug reports will obviously not be up-to-date in Bugzilla.
Bug 11103 - Lucene search 2 uses GNU-specific cp options when snapshotting indexes
Lucene search 2 uses GNU-specific cp options when snapshotting indexes
Product: Wikimedia
Classification: Unclassified
lucene-search-2 (Other open bugs)
All All
: Normal enhancement (vote)
: ---
Assigned To: Nobody - You can work on this!
Depends on:
  Show dependency treegraph
Reported: 2007-08-28 20:49 UTC by Brion Vibber
Modified: 2013-03-26 11:25 UTC (History)
2 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Description Brion Vibber 2007-08-28 20:49:23 UTC
Testing a local copy of the lucene-search-2 module on my MacBook, I noticed this error dropping out while building the indexes:

2383 [main] INFO  org.wikimedia.lsearch.index.IndexThread  - Making snapshot for trunkwiki
2412 [main] WARN  org.wikimedia.lsearch.util.Command  - Got exit value 64 while executing /bin/cp -lr /opt/web/lsearch/import/trunkwiki/_8t.cfs /opt/web/lsearch/snapshot/trunkwiki/20070828164041/_8t.cfs
2414 [main] ERROR org.wikimedia.lsearch.index.IndexThread  - Error making snapshot /opt/web/lsearch/snapshot/trunkwiki/20070828164041: Error executing command: cp: illegal option -- lusage: cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src target       cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src1 ... srcN directory

The -l option to cp seems to be a GNU extension, and isn't present in the Mac OS X version of cp. Needless to say, /bin/cp may also not be present on non-Unix systems. ;)

cp -lr is also used in UpdateThread, as is rm -rf and /usr/bin/rsync (if rsync is in use).

It might be wise to make the snapshots in a more portable way, using portable file-manipulation functions. The rsync path should be customizable; in theory it might be nice if it runs on Windows as well as name-brand *nixes.
Comment 1 Robert Stojnic 2007-08-29 00:28:27 UTC
Currently lsearch requires linux. It could work on MacOS since it 
afaik support hard and soft links, with some minor modification. 

Windows is more difficult, one would probably need something like
info files that would describe the links, or just use a slower 
copy, instead of linking. 

So, the above cp would be: cp + ln. Hopefully, in some next release
it should have customizable paths, and should be able to detect
the OS, and use the most efficient solution for it. 

Comment 2 Brion Vibber 2007-08-29 20:08:07 UTC
Creating the destination directory, then shelling out to ln to link each file should be compatible with any UNIXy system (Linux, Mac OS X, *BSD, Solaris, etc) and not a very difficult change, so that'd be my recommendation for a short-term change.

Windows does support hardlinks of some sort on NTFS, though I couldn't tell you if it's worth the bother. :)

I did a little googling yesterday for Java code supporting creation of hardlinks; this util class from an Apache project has a function which allegedly can create hardlinks on either Unix or Windows by shelling out the appropriate low-level command:

Haven't tested it meself. :)
Comment 3 Rob Church 2007-08-29 20:11:45 UTC
(In reply to comment #2)
> Windows does support hardlinks of some sort on NTFS, though I couldn't tell you
> if it's worth the bother. :)

NTFS Junction Points? No, it's not worth the bother, and there's no standard toolkit to create or manage them - Windows uses them internally, and exposes them via Win32, but doesn't ship with any kind of command-line utility comparable to ln.
Comment 4 Brion Vibber 2007-08-29 20:18:01 UTC

C:\Documents and Settings\brion> fsutil hardlink create bar.txt foo.txt
Hardlink created for C:\Documents and Settings\brion\bar.txt <<===>> C:\Documents and Settings\brion\foo.txt

Comment 5 Rob Church 2007-08-29 20:27:34 UTC
Well, in my defence, what I said was more or less accurate; that's not an NTFS Junction Point at all. What's actually indefensible here is that I read "hardlink" and thought of the (dodgy) equivalent of a symlink.

Points off all round.
Comment 6 Robert Stojnic 2007-10-02 23:15:46 UTC
Fixed in r26333. 
Comment 7 Andre Klapper 2013-03-26 11:25:13 UTC
[Merging "MediaWiki extensions/Lucene Search" into "Wikimedia/lucene-search2", see bug 46542. You can filter bugmail for: search-component-merge-20130326 ]

Note You need to log in before you can comment on or make changes to this bug.