Last modified: 2012-08-04 21:11:13 UTC
Here are our requirements for AFT Metrics Stage 5, which will measure usage of the final feedback form (Option 6) -- as compared to the current feedback form (Option 1). Our plan for this Stage 5 test is posted here: https://docs.google.com/a/wikimedia.org/document/d/1YzWs0gsPeDiURzS6UZ1r_eK-KVsyy5ieoKowNcc3ExM/edit To collect data for this test, we plan to to bucket users in two groups: * 50% of AFT5 visitors will see the new Option 6 feedback form (when they click on 'Post feedback', they will see the CTA 3 inviting them to a survey about that new form, as outlined here: http://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#CTA_3:_Take_a_Survey) * 50% AFT5 will see the current Option 1 form (when they click on 'Post feedback', they will see the CTA 5 that invites them to view the feedback page, as outlined here: http://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#CTA_5:_View_feedback). As proposed by Reha, we plan to re-use the clicktracking infrastructure that is already in place for AFT5, by changing the experimental prefix so that they're easily distinguished between the two forms: * M5_1 (for current Option 1 feedback form) * M5_6 (for the new Option 6 feedback form) Metrics Test 3 experimental prefixes are: SE_1X, SE_1E, SE_4X_edit, SE_4X_learn-more, and SE_4E_edit. (Although currently only SE_1X is being logged, because the others have all been turned off.) For Metrics Test 5, we will simply change the experimental prefixes from Metrics Test 3 to M5_1 and M5_6 and keep the same events from Metrics test 3 that are relevant to this test 5 (e.g.: M5_1_edit, M5_1_learn-more, M5_6_edit, M5_6_learn-more). P.S.: If easy to do, it would also be helpful to track how often people click on the 'Yes' or 'No' buttons for both forms, to determine if we are losing traffic due to the fact that some people may not post feedback for the new feedback form Option 6. If that is feasible for this deployment, then the events might include: M5_1_yes, M5_1_no, M5_6_yes, M5_6_no. If this is not feasible in this timeframe, please hold off on this section of this ticket. See also these related Bugzilla tickets: * Bug 36647 - Article Feedback - Final Feedback Form (Option 6) https://bugzilla.wikimedia.org/show_bug.cgi?id=36647 * Bug 37262 - Article Feedback - CTA 5 Tweaks https://bugzilla.wikimedia.org/show_bug.cgi?id=37262 * Bug 36650 - Metrics Stage 4 Test https://bugzilla.wikimedia.org/show_bug.cgi?id=36650
Pushed to prototype and submitted to gerrit: https://gerrit.wikimedia.org/r/#/c/10131/
For our next deployment on June 12, let's make these changes to the metrics stage 5 test: • disable CTA3 ("take a survey"), as we have collected enough data for the survey • activate CTA5 ("view feedback page") for all users (including Option 6 users) • disable click tracking for the Option 6 feedback form on June 12 (we will have enough data on impressions since June 5) • disable click tracking for section edit links / edit tabs (we don't need this data any more as the Stage 3 analysis is completed) • leave the 2 buckets active (Option1 and Option6) beyond June 12 (in case we need more data to compare the volume of feedback between the two designs.) • for metrics stage 4, leave click tracking active for the Feedback Page past June 12 (as it's generating a limited amount of data in the log and we still want to measure its usage)
Made changes and submitted to gerrit: https://gerrit.wikimedia.org/r/10942
