Last modified: 2013-12-29 22:10:41 UTC
DjVu image files may contain a machine-readable form of the text contained in the represented document. It could be useful to extract this text and include it in the search index for the file's description page. It's probably possible to extract text using the DjVuLibre library or tools.
Do you mean that the file's description page should be on the search results of the 'extracted text' of the DjVu file ?
I'm pretty sure he means that search results should include hits from within the text layer of the file scans proper. So if the scans include text "Foobar" a search for "bar" would return the text from the scan's text layer as a result even though the text layer had not been extracted and placed on a wiki page. There's no reason that this should be restricted to DjVu's (though there was at the time the bug was filed)
As I can see the bug here is quite old and additionally marked "low" in prority. Is this bug up to be fixed at all? In my opinion to solve this bug here is a *must have*. DrTrigonBot [1] does file content based categorization in commons. Due to this embedded text from PDF (later DJVU too) is extracted and processed. We are currently debating [2] about whether to store this text data to a page - in order to enable the mediawiki search engine to index and find those contents - or not. Now the question is: When is this bug scheduled to become fixed? Will it be fixed at all? IF NOT; As mentioned DrTrigonBot could dump the files text content to a dedicated page in order to enable the mediawiki search engine to handle them. This should be considered as a work-a-round only and would not be needed at all, if and when this bug here is solved. [1] http://commons.wikimedia.org/wiki/User:DrTrigonBot [2] http://commons.wikimedia.org/wiki/User_talk:DrTrigonBot/JavaScript#PDF_content_extraction
I doubt this will be solved any time soon, there's nobody working on this or related issues and search is a monster nobody really wants to touch AFAIK, so I'd suggest you to implement whatever workaround you think it's worth about 5 more years of usage. The pages you create should probably be as hidden as possible to users, in particular they shouldn't be indexed by external search engines or they would e.g. "compete" with Wikisource (or even archive.org) which doesn't make any sense.
Is it possible to include metadata into the search (indexing) in mediawiki software? As I was informed the text layer gets extracted by Pdf Handler [1] and is stored in the images (PDFs) metadata [2] (name="0"). [1] https://www.mediawiki.org/wiki/Extension:PdfHandler [2] http://commons.wikimedia.org/w/api.php?action=query&iilimit=500&iiprop=metadata|timestamp&prop=imageinfo&titles=File:Resume-.pdf
Change 101252 had a related patch set uploaded by Brian Wolff: Begin indexing file text from pdf/djvu files https://gerrit.wikimedia.org/r/101252
(In reply to comment #6) > Change 101252 had a related patch set uploaded by Brian Wolff: > Begin indexing file text from pdf/djvu files > > https://gerrit.wikimedia.org/r/101252 Ah, wonderful. That's in CirrusSearch and the core part was already done in https://gerrit.wikimedia.org/r/#/c/99715/ , but there's nothing more specific than the DjVu component so I'm not moving this bug.
Change 101252 merged by jenkins-bot: Index and search file text from pdf/djvu files https://gerrit.wikimedia.org/r/101252
Merged. It won't take effect until a full reindex of everything in the file namespace. That'll take a few days after the deployment. Results will start showing up when the document is indexed. Also, the file text results are with .8 of a page text result from a scoring standpoint. Finally, this'll work with any files from which mediawiki is able to extract text. If a new file type is plugged in at a later date those files will have to be reindexed for the text to be searchable.
Nice! Good job - thanks! What about including metadata into the search (indexing) as well??
(In reply to comment #10) > Nice! Good job - thanks! > > What about including metadata into the search (indexing) as well?? Not a bad idea. We have a bug filed for it somewhere?
(In reply to comment #11) > Not a bad idea. We have a bug filed for it somewhere? What about bug 21061, may be bug 13370 as well?! Or shall I create a new one?
(In reply to comment #12) > (In reply to comment #11) > > Not a bad idea. We have a bug filed for it somewhere? > > What about bug 21061, may be bug 13370 as well?! Or shall I create a new one? Those will do great :)
Good! I linked them. Am I wrong or should we now be able to close bug 6422 as well?
