Last modified: 2014-11-20 13:34:25 UTC
I just tried to upload http://tools.wikimedia.pl/~odder/whitehouse/41d5129de542285b6c62.webm using https://commons.wikimedia.org/wiki/Special:Upload as well as via API (I have "upload_by_url" userright enabled). Request URL: https://commons.wikimedia.org/w/api.php?action=upload&format=json POST data from API sandbox: filename=Test%20test.webm&comment=Test%20bug%2072897&url=http%3A%2F%2Ftools.wikimedia.pl%2F~odder%2Fwhitehouse%2F41d5129de542285b6c62.webm&token=d27ea41fb633146835d1a2d1e962f37e545f9bd9%2B%5C Result: { "servedby": "mw1139", "error": { "code": "http-bad-status", "info": "Error fetching file from remote source", "0": "403", "1": "Forbidden", "*": "See https://commons.wikimedia.org/w/api.php for API usage" } } Log entry on the tools.wikimedia.pl server: tools.wikimedia.pl 208.80.xxx.yyy - - [09/Nov/2014:18:03:30 +0100] "GET /~odder/whitehouse/41d5129de542285b6c62.webm HTTP/1.1" 200 932365538 "-" "MediaWiki/1.25wmf6" so MediaWiki gets 200 from the origin server
Maybe check to see if it works with a small test file, just to rule out the size of the file triggering something (Yeah, I know, 403 would be the wrong error code for file too big, but just in case).
I'm idly wondering whether you're setting a User-Agent and whether you can perform other actions such as edits.
From the url-downloader config: reply_body_max_size 534773760 allow all Looks like upload by url has a max file size of 510 mb. (Also I feel like even without that, things might time out before a 900 mb video got uploaded)
(In reply to MZMcBride from comment #2) Ignore me! I mis-read the bug report and Bawolff set me straight.
(In reply to Bawolff (Brian Wolff) from comment #3) > From the url-downloader config: > > reply_body_max_size 534773760 allow all > > Looks like upload by url has a max file size of 510 mb. (Also I feel like > even without that, things might time out before a 900 mb video got uploaded) https://git.wikimedia.org/blob/operations%2Fpuppet.git/df4b24132abb856323e69a0decf0ebcde65b67fa/modules%2Furl_downloader%2Ftemplates%2Fsquid.conf.erb#L81 Yeah, there are a few references to 510 MB in there.
Change 172120 had a related patch set uploaded by Brian Wolff: Increase max file size of url downloader proxy to 1010mb https://gerrit.wikimedia.org/r/172120
Change 172120 merged by Faidon Liambotis: Increase max file size of url downloader proxy to 1010mb https://gerrit.wikimedia.org/r/172120
Puppet patch has been merged.
If the problem was indeed the file being too large, the API should have a less misleading way of reporting it, so I don't think this is fully fixed. Marcin, can you verify whether the upload works now?
Right now I get "HTTP request timed out.", so we are one stage ahead. It didn't wait very long before timeout, few seconds maybe.
In reply to Tisza Gergő from comment #9) > If the problem was indeed the file being too large, the API should have a > less misleading way of reporting it, so I don't think this is fully fixed. > Its a proxy squid server in between the api and the rest of the internet. I dont think mw has any way of knowing it hit a file too big instead of legit 403. (In reply to Marcin Cieślak from comment #10) > Right now I get "HTTP request timed out.", so we are one stage ahead. It > didn't wait very long before timeout, few seconds maybe. I think the timeout is currently 30 seconds(?) Im not surprised you hit it. You would have to have some pretty good conectivity to transfer a 900mb file in 30 seconds.
Change 172437 had a related patch set uploaded by saper: More debug diagnostics for upload by URL https://gerrit.wikimedia.org/r/172437
The only place where squid (or the target website) tell us more details is the body of the error response. Gerrit change #172437 adds remote server output to the debug log to help troubleshoot such issues.
(In reply to Bawolff (Brian Wolff) from comment #11) > Its a proxy squid server in between the api and the rest of the internet. I > dont think mw has any way of knowing it hit a file too big instead of legit > 403. We are talking about a squid operated by WM-PL, right? AFAIK wikimedia.pl is not WMF-hosted. That would make this a bug in the squid configuration of WM-PL, presumably. Is there a tracker where this could be upstreamed then? > I think the timeout is currently 30 seconds(?) Im not surprised you hit it. > You would have to have some pretty good connectivity to transfer a 900mb file > in 30 seconds. The deeper issue is, then, that we don't have a method for upload-by-url that's not limited by the timeout (unlike normal uploads, where we have the chunked upload API)?
(In reply to Tisza Gergő from comment #14) > (In reply to Bawolff (Brian Wolff) from comment #11) > > Its a proxy squid server in between the api and the rest of the internet. I > > dont think mw has any way of knowing it hit a file too big instead of legit > > 403. > > We are talking about a squid operated by WM-PL, right? AFAIK wikimedia.pl is > not WMF-hosted. No, we are talking about url-downloader.wikimedia.org proxy. I believe the proxy is used as a security measure (not 100% sure against what, but isolating user trigerrable http downloads is probably just a goodvidea in general) > The deeper issue is, then, that we don't have a method for upload-by-url > that's not limited by the timeout (unlike normal uploads, where we have the > chunked upload API)? Yes. We used to have an async mode, but its disabled and possibly has issues (not sure what exactly). We may be able to ease issue by increasing various timeouts or removing the restriction against using tool labs (which afaik has a really fast link to production, being in same data centre and all), but ultimately to truly fix this we would need to reserurecte async mode (otoh i have no idea how that would tie in with current UI. But current ui sucks a lot so i suppose it wouldnt matter much)
