Last modified: 2013-03-26 11:25:13 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T13103, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 11103 - Lucene search 2 uses GNU-specific cp options when snapshotting indexes
Lucene search 2 uses GNU-specific cp options when snapshotting indexes
Status: RESOLVED FIXED
Product: Wikimedia
Classification: Unclassified
lucene-search-2 (Other open bugs)
unspecified
All All
: Normal enhancement (vote)
: ---
Assigned To: Nobody - You can work on this!
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2007-08-28 20:49 UTC by Brion Vibber
Modified: 2013-03-26 11:25 UTC (History)
2 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Brion Vibber 2007-08-28 20:49:23 UTC
Testing a local copy of the lucene-search-2 module on my MacBook, I noticed this error dropping out while building the indexes:

2383 [main] INFO  org.wikimedia.lsearch.index.IndexThread  - Making snapshot for trunkwiki
2412 [main] WARN  org.wikimedia.lsearch.util.Command  - Got exit value 64 while executing /bin/cp -lr /opt/web/lsearch/import/trunkwiki/_8t.cfs /opt/web/lsearch/snapshot/trunkwiki/20070828164041/_8t.cfs
2414 [main] ERROR org.wikimedia.lsearch.index.IndexThread  - Error making snapshot /opt/web/lsearch/snapshot/trunkwiki/20070828164041: Error executing command: cp: illegal option -- lusage: cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src target       cp [-R [-H | -L | -P]] [-f | -i | -n] [-pv] src1 ... srcN directory

The -l option to cp seems to be a GNU extension, and isn't present in the Mac OS X version of cp. Needless to say, /bin/cp may also not be present on non-Unix systems. ;)

cp -lr is also used in UpdateThread, as is rm -rf and /usr/bin/rsync (if rsync is in use).

It might be wise to make the snapshots in a more portable way, using portable file-manipulation functions. The rsync path should be customizable; in theory it might be nice if it runs on Windows as well as name-brand *nixes.
Comment 1 Robert Stojnic 2007-08-29 00:28:27 UTC
Currently lsearch requires linux. It could work on MacOS since it 
afaik support hard and soft links, with some minor modification. 

Windows is more difficult, one would probably need something like
info files that would describe the links, or just use a slower 
copy, instead of linking. 

So, the above cp would be: cp + ln. Hopefully, in some next release
it should have customizable paths, and should be able to detect
the OS, and use the most efficient solution for it. 

Comment 2 Brion Vibber 2007-08-29 20:08:07 UTC
Creating the destination directory, then shelling out to ln to link each file should be compatible with any UNIXy system (Linux, Mac OS X, *BSD, Solaris, etc) and not a very difficult change, so that'd be my recommendation for a short-term change.

Windows does support hardlinks of some sort on NTFS, though I couldn't tell you if it's worth the bother. :)

I did a little googling yesterday for Java code supporting creation of hardlinks; this util class from an Apache project has a function which allegedly can create hardlinks on either Unix or Windows by shelling out the appropriate low-level command:

http://svn.apache.org/viewvc/lucene/hadoop/trunk/src/java/org/apache/hadoop/fs/FileUtil.java?view=markup

Haven't tested it meself. :)
Comment 3 Rob Church 2007-08-29 20:11:45 UTC
(In reply to comment #2)
> Windows does support hardlinks of some sort on NTFS, though I couldn't tell you
> if it's worth the bother. :)

NTFS Junction Points? No, it's not worth the bother, and there's no standard toolkit to create or manage them - Windows uses them internally, and exposes them via Win32, but doesn't ship with any kind of command-line utility comparable to ln.
Comment 4 Brion Vibber 2007-08-29 20:18:01 UTC
*ahem*

C:\Documents and Settings\brion> fsutil hardlink create bar.txt foo.txt
Hardlink created for C:\Documents and Settings\brion\bar.txt <<===>> C:\Documents and Settings\brion\foo.txt

:)
Comment 5 Rob Church 2007-08-29 20:27:34 UTC
Well, in my defence, what I said was more or less accurate; that's not an NTFS Junction Point at all. What's actually indefensible here is that I read "hardlink" and thought of the (dodgy) equivalent of a symlink.

Points off all round.
Comment 6 Robert Stojnic 2007-10-02 23:15:46 UTC
Fixed in r26333. 
Comment 7 Andre Klapper 2013-03-26 11:25:13 UTC
[Merging "MediaWiki extensions/Lucene Search" into "Wikimedia/lucene-search2", see bug 46542. You can filter bugmail for: search-component-merge-20130326 ]

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links