Last modified: 2012-11-01 12:28:13 UTC
We would like to study the impact of AFTv5 to answer these key research questions, using new data: 1. Editor Conversions: how many readers who post feedback go on to become editors? 2. New Editor Productivity: how many of these new editors edit productively? (with same reversion rates as other new editors) Previous research showed that, when the CTA is present, a number of readers go on to make productive edits, , as outlined in this report: http://meta.wikimedia.org/wiki/Research:Article_feedback/Stage_3/Conversion_and_newcomer_quality We would now like to collect more data for at least one week to verify if this is still the case, now that AFT5 has been deployed to 10% (with a new feedback form and more sophisticated monitoring tools). To that end, we would like to conduct the same Metrics Stage 3 test that we ran from 4/27 to 5/07, when clicks from CTA 1 (Edit this page) were gathered from the clicktracking logs. For this new study 6, we have created a task card and checklist on Trello to track our progress: https://trello.com/card/aft5-edit-cta-conversions-using-current-data/5069e3859c41bc8f712411a3/4 From a development standpoint, here are some of the main action items that are needed for this test: a) change the bucketing parameters to show the CTA1 at 100% for all readers who have not edited before (but not to existing editors) b) re-enable clicktracking to collect fresh data, tracking these metrics for non-editors: - total impressions of the CTA1 (Edit this page) - total clicks on the CTA1 (causing the Edit form to appear) - total clicks on the Edit Save button c) re-enable similar clicktracking for the Page Edit (and Section Edit) buttons, so we can compare the activity of users who edit through the AFT5 CTA1 with users who edit through conventional means. d) re-enable the special tag that was associated with edits in Metrics Stage 3, so that we can track the number of saved edits which were reverted, and determine the productivity of these edits, compared with other new editors who click on the regular Edit button. e) test thoroughly that we are collecting the proper data and tagging edits correctly on prototype before deployment. Dario will provide more details in comments below on what exact codes and tags we want to use to track this properly. See also section 1 of this Etherpad page for more details on our overall pre-rampup data collection plan: http://etherpad.wikimedia.org/AFT5PreRampUpData For more info on our CTAs and how AFT5 works, check this feature requirements page: http://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#CTA_1:_Edit_this_page See also related bugs from earlier efforts on this topic: https://bugzilla.wikimedia.org/show_bug.cgi?id=35421 https://bugzilla.wikimedia.org/show_bug.cgi?id=38019
Looks good to me, the only missing part is save-attempt events vs save-complete events for the different edit funnels (CTA1, Section Edit, Tab Edit). I've updated the specs on this page, based on Stage 3 events: http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#CTA1_update_.28October_2012.29 Let me know if you have any question
Dario: pushed to Gerrit (https://gerrit.wikimedia.org/r/#/c/26782/) & prototype. Any change you can test in on prototype already. There's a few changes to the events you proposed at http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#CTA1_update_.28October_2012.29 * the "-bottom" suffix is removed from all events, since - with the disappearance of the overlay - bottom is all there is left * cta_learn_more will not be shown if it's at 0%, not even as fallback - maybe we should set the percentages to 99% (CTA1) - 1% (CTA2)? In more detail: option6X-init (1%) -> OK option6X-noedit-init (1%) -> OK - note: users tracked with this one do not also get option6X-init option6X-impression-bottom -> changed to option6X-impression (there is no more overlay form, so -bottom suffix makes no longer sense) - is also at 1% option6X-submit-bottom -> changed to option6X-submit_attempt (followed by option6X-submit_success or option6X-submit_error_<error>) option6X-cta_edit-impression-bottom -> changed to option6X-cta_edit-impression option6X-cta_learn_more-impression-bottom -> This one will not trigger: if a CTA is bucketed at 0%, it can not even be shown as fall-back CTA. Maybe do 99%-1%? option6X-cta_edit-button_click-bottom -> changed to option6X-cta_edit-button_click option6X-cta_learn_more-button_click-bottom -> This one will not trigger: if a CTA is bucketed at 0%, it can not even be shown as fall-back CTA. Maybe do 99%-1%? option6X-cta_edit-edit_attempt-bottom -> changed to option6X-cta_edit-edit_attempt option6X-cta_edit-edit_success-bottom -> changed to option6X-cta_edit-edit_success option6X-edit_tab_link-click -> OK option6X-section_edit_link-click -> OK option6X-view_source_tab_link-click -> OK option6X-edit_tab_link-edit_attempt -> OK option6X-section_edit_link-edit_attempt -> OK option6X-edit_tab_link-edit_success -> OK option6X-section_edit_link-edit_success -> OK
This looks good, except that I'm not sure what you mean when you say: option6X-cta_learn_more-button_click-bottom -> This one will not trigger: if a CTA is bucketed at 0%, it can not even be shown as fall-back CTA. Maybe do 99%-1%? and option6X-cta_learn_more-impression-bottom -> This one will not trigger: if a CTA is bucketed at 0%, it can not even be shown as fall-back CTA. Maybe do 99%-1%? I'm under the impression that we'd like the "learn more" CTA to be a fallback only when the reader does not have the rights to edit the page. Does your note state that we'd have to show the "learn more" CTA to some readers who *do* have rights to edit in order to allow it to be a fallback in the cases where readers do not? If so, that seems ... odd.
Well if CTA1 (edit) is at 100%, CTA2 (learn more) will be at 0%. CTA's bucketed at 0% are never - not even as fallback - displayed & changing that would be quite a lot of work. I'd suggest enabling CTA1 at 99% (effectively receiving 99% - <protected pages>) & CTA2 at 1% (effectively receiving 1% + <protected pages>)
I recommend that we run CTA1 at 100% and drop CTA2 entirely. On semiprotected pages there will simply be no fallback CTA, which is fine as we want to focus on anonymous edit activity for editable pages only. Let me know if you have any concerns with this.
I also requested that we switch from the CT session to the persistent anon token set by mw.user.id(), which will allow us to get clickthrough/conversion data about recurrent edits from the same client across sessions. Matthias noted that it's trivial to make this change before we push this to production on Thursday.
"I recommend that we run CTA1 at 100% and drop CTA2 entirely." I would recommend against dropping CTA2. While we may not focus on CTA2 for this metrics study, we'll miss a chance to reach out to those editors that fill out feedback on a protected page in this timeframe (they'll only see " Thanks! Your post can be viewed on this feedback page.") - there probably will only be very few though. Would this interfere with gathering accurate data? Likewise, CTA4 will need to be enabled as well (or anon users won't get any CTA) - this won't mess with the percentage of users that are getting CTA1, but you'll be seeing option6X-cta_signup_login-XYZ as well, though I'm assuming this won't be a problem? "I also requested that we switch from the CT session to the persistent anon token set by mw.user.id()" Done & pushed to prototype - could you verify this works as expected?
Final setup, as agreed on IRC: * CTA1 (Edit) at 100% for anons * CTA6 (Visit Teahouse) at 100% for registered (as this is not affecting anons) * CTA2 (Learn More) and CTA4 (Signup) entirely disabled On semiprotected pages no CTA will be served to anons.
This is live on testwiki, I tested and looked up the log and saw a number of issues, see sample below testwiki ext.articleFeedbackv5@10-option6X_-cta_edit-button_click 20121011200909 0 x38YiXpxrvF17aqUxE6Btv8mF4gFjcSG 0 0 0 0 0 Golden-crowned_Sparrow|147432 testwiki ext.articleFeedbackv5@10-option6X_-cta_edit-edit_attempt-bottom 20121011200921 0 x38YiXpxrvF17aqUxE6Btv8mF4gFjcSG 0 0 0 0 0 Golden-crowned Sparrow|147432 testwiki ext.articleFeedbackv5@10-option6X_-cta_edit-edit_attempt-bottom 20121011200933 0 x38YiXpxrvF17aqUxE6Btv8mF4gFjcSG 0 0 0 0 0 Golden-crowned Sparrow|147432 testwiki ext.articleFeedbackv5@10-option6X_-cta_edit-edit_success-bottom 20121011200936 0 x38YiXpxrvF17aqUxE6Btv8mF4gFjcSG 0 0 0 0 0 Golden-crowned Sparrow|147436 High priority: * All impression events (both option6X-impression and option6X-cta_edit-impression-bottom) are missing and they are critical for the metrics we need to capture Lower priority: * there is an underscore in the bucket name that should be removed (option6X_) * the "-bottom" suffix is still there, while according to the comments above it should have gone
Anon token is now logged as requested instead of the clicktracking session and the cookie reset if missing, thanks.
section edit and tab edit events logged as expected, thanks.
Fixed all issues, in production as we speak
Re-opening this ticket, because of issues associated with the data we collected, according to Dario: On Oct 23, 2012, at 3:06 PM, Dario Taraborelli wrote: Fabrice, after reviewing the problem with Aaron, Matthias and S, we decided to unstage the AFT changes and hold off with the enwiki deployment until we've fixed it. The problem is caused by the use of inconsistent tokens in the CT log (the clicktracking session, i.e. CT's default, for some events, the anonymous token generated by mw.user.id() for other events). Where we are now: • Aaron confirmed that the CTA1 data we collected last week is affected by this bug (this applies both to CTA edits and regular edits). He'll report further issues with the data on this thread. • S and Matthias went ahead and deployed the campaign support in ACUX, which has already been fully tested and QA'd. This means that whenever we're ready to run the CTA4 again, we'll be able to log events straightaway • I recommend that we review both the code and the data and target Thursday for a deployment of CTA1 at .5 and CTA4 at .5. This should give us enough data for the basic metrics we were aiming to get and allow us to decommission CT by 10/30 as originally planned. Let me know if you have any concerns with this plan. Thanks everybody (and Matthias in particular) for helping nailing down this problem. Dario
