Last modified: 2011-06-14 18:21:10 UTC

Wikimedia Bugzilla is closed!

Wikimedia migrated from Bugzilla to Phabricator. Bug reports are handled in Wikimedia Phabricator.
This static website is read-only and for historical purposes. It is not possible to log in and except for displaying bug reports and their history, links might be broken. See T31184, the corresponding Phabricator task for complete and up-to-date bug report information.
Bug 29184 - Mysterious failure in 720p WebM transcode
Mysterious failure in 720p WebM transcode
Status: RESOLVED FIXED
Product: MediaWiki extensions
Classification: Unclassified
TimedMediaHandler (Other open bugs)
unspecified
All All
: Unprioritized normal (vote)
: ---
Assigned To: Michael Dale
:
Depends on:
Blocks: 27699
  Show dependency treegraph
 
Reported: 2011-05-28 00:29 UTC by Brion Vibber
Modified: 2011-06-14 18:21 UTC (History)
2 users (show)

See Also:
Web browser: ---
Mobile Platform: ---
Assignee Huggle Beta Tester: ---


Attachments

Description Brion Vibber 2011-05-28 00:29:34 UTC
Transcoding a local copy of the 1920x1088 test file from http://prototype.wikimedia.org/tmh/images/b/b2/File-Arborophila_brunneopectus_pair_feeding_-_Kaeng_Krachan.ogv

Ubuntu 11.04 x86_64 system with ffmpeg stock 0.6.2-4:0.6.2-1ubuntu1.

The '480p' WebM file renders fine, but the '720p' WebM version fails. The .queue.webm file got written out to about 37 of the 42 seconds of input data, and is missing index etc.


The recorded 'error' looks like regular output:

*************************** 5. row ***************************
            transcode_id: 20
    transcode_image_name: File-Arborophila_brunneopectus_pair_feeding_-_Kaeng_Krachan.ogv
           transcode_key: 480_900kbs.webm
         transcode_error: 
   transcode_time_addjob: 2011-05-28 00:03:06
transcode_time_startwork: 2011-05-28 00:09:05
  transcode_time_success: 2011-05-28 00:10:48
    transcode_time_error: NULL
 transcode_final_bitrate: 888048
*************************** 6. row ***************************
            transcode_id: 21
    transcode_image_name: File-Arborophila_brunneopectus_pair_feeding_-_Kaeng_Krachan.ogv
           transcode_key: 720_VBR.webm
         transcode_error: FFmpeg version 0.6.2-4:0.6.2-1ubuntu1, Copyright (c) 2000-2010 the Libav developers
  built on Mar 22 2011 15:55:04 with gcc 4.5.2
  configuration: --extra-version=4:0.6.2-1ubuntu1 --prefix=/usr --enable-avfilter --enable-avfilter-lavf --enable-vdpau --enable-bzlib --enable-libgsm --enable-libschroedinger --enable-libspeex --enable-libtheora --enable-libvorbis --enable-pthreads --enable-zlib --enable-libvpx --disable-stripping --enable-runtime-cpudetect --enable-vaapi --enable-gpl --enable-postproc --enable-swscale --enable-x11grab --enable-libdc1394 --enable-shared --disable-static
  libavutil     50.15. 1 / 50.15. 1
  libavcodec    52.72. 2 / 52.72. 2
  libavformat   52.64. 2 / 52.64. 2
  libavdevice   52. 2. 0 / 52. 2. 0
  libavfilter    1.19. 0 /  1.19. 0
  libswscale     0.11. 0 /  0.11. 0
  libpostproc   51. 2. 0 / 51. 2. 0
Input #0, ogg, from '/var/www/trunk/images/b/b2/File-Arborophila_brunneopectus_pair_feeding_-_Kaeng_Krachan.ogv':
  Duration: 00:00:42.20, start: 0.000000, bitrate: 12632 kb/s
    Stream #0.0: Data: skeleton
    Stream #0.1: Video: theora, yuv420p, 1920x1088, 25 fps, 25 tbr, 25 tbn, 25 tbc
    Stream #0.2: Audio: vorbis, 48000 Hz, stereo, s16, 192 kb/s
    Metadata:
      ENCODER         : ffmpeg2theora-0.26+svn16924
      SOURCE_OSHASH   : cd67a9923275c17c
[libvpx @ 0x15ae000]v0.9.6
Output #0, webm, to '/var/www/trunk/images/thumb/b/b2/File-Arborophila_brunneopectus_pair_feeding_-_Kaeng_Krachan.ogv/File-Arborophila_brunneopectus_pair_feeding_-_Kaeng_Krachan.ogv.720_VBR.webm.queue.webm':
  Metadata:
    encoder         : Lavf52.64.2
    Stream #0.0: Video: libvpx, yuv420p, 1280x725, q=19-19, 200 kb/s, 1k tbn, 25 tbc
    Stream #0.1: Audio: libvorbis, 48000 Hz, stereo, s16, 64 kb/s
    Metadata:
      ENCODER         : ffmpeg2theora-0.26+svn16924
      SOURCE_OSHASH   : cd67a9923275c17c
Stream mapping:
  Stream #0.1 -> #0.0
  Stream #0.2 -> #0.1
Press [q] to stop encoding
frame=  945 fps= 10 q=0.0 size=   22121kB time=37.59 bitrate=4821.1kbits/s    
   transcode_time_addjob: 2011-05-28 00:03:06
transcode_time_startwork: 2011-05-28 00:07:31
  transcode_time_success: NULL
    transcode_time_error: 2011-05-28 00:09:05
 transcode_final_bitrate: 0


Using these shell memory settings to override the defaults:

$wgMaxShellMemory = 1024000;
$wgMaxShellFileSize = 1024000;

Possibly these are not enough and ffmpeg is dying; README recommends 102400*64 for $wgMaxShellMemory, though there are no recommendations on the wiki page.

Is there any way we can detect a failed process specifically?
Comment 1 Michael Dale 2011-06-02 02:02:38 UTC
it does look like a good encode from the error... It does have a high bitrate ... Maybe we need to recommend more memory for ffmpeg?

Can you upload or put this asset somewhere so Jan and I can take a look at it? 

Jan is there a good target limit for memory that ffmpeg can consume when dealing with normal range of HD content? 

The 64Megs recommendation is not based in science of any kind.
Comment 2 Michael Dale 2011-06-14 18:21:10 UTC
I have added a new features help with this:
 
1) We list transcode status and error on the image page, that would show why the transcode failed. 
2) We use the --maxtime argument when calling the runJobs.php script and with r90068 we enforce a maxtime for transcodes.  
3) In r90068 I have made transcode calls run in the background and added a check for  growing file function that checks if the target file is growing. If the target file stops growing we kill the process and flag the transcode as an error. This should make it a lot harder for frozen transcodes to stay around for any substantial amount of time.

Note You need to log in before you can comment on or make changes to this bug.


Navigation
Links