Last modified: 2012-11-05 17:56:50 UTC
Now that the AFT5 Metrics Studies 6 have ended, we would like to remove some of the clicktracking and change the CTA4 and overall bucketing, as proposed below: 1. End the CTA1 metrics test that measures edit conversions, as described in Bugzilla ticket 40775: https://bugzilla.wikimedia.org/show_bug.cgi?id=40775 2. End the CTA4 metrics test that measures signups, as described in Bugzilla ticket 41180: https://bugzilla.wikimedia.org/show_bug.cgi?id=41180 BUT: Keep passing the AFT5 campaign code in CTA4 when sending folks to ACUX, so we can keep tracking AFT5 signups for future engagement programs. 3. Modify CTA4 so that the Login button is turned into a simple text link, to discourage clicking on it (same style as 'Maybe later', which should be removed, along with the word 'or'). Once E3 creates a special version of the AFT5 signup page with a more prominent Login button, we can remove that link altogether. Here's the current spec for CTA4, for reference: http://www.mediawiki.org/wiki/Article_feedback/Version_5/Feature_Requirements#CTA_4:_Create_an_account_or_login 4. Change the bucketing of the CTAs as follows: 0% CTA1 (Edit this page), 0% CTA2 (Learn more), 80% CTA4 (Signup), 10% CTA5 (Feedback page), 10% CTA6 (Teahouse) (I think we should keep sending some traffic to the feedback page and the Teahouse for now, which is why I kept these settings the same as before, with the rest of the traffic going to signups). Note that for logged in editors, we will continue to show CTA5 at 100%, to encourage them to moderate the feedback page. And if a page is semi-protected and the user is logged in, we should now send them to CTA5 (instead of CTA2).
Re: 4. we have no plans for further funnel analysis but we definitely want to analyze post-registration behavior for the CTA4 cohort (and the cohort is still pretty thin). If this is the goal, I recommend keeping CTA4 (Signup) maxed out at 100% for anons (which we haven't tested yet). CTA5 (Visit feedback page) at 10% is neither going to produce usable data nor any visible impact on moderation activity and I don't quite see the benefits of running it. Also note that CTA6 (Teahouse) is currently displayed to registered users only. I also want to clarify that by disabling clicktracking for funnel analysis we won't be able to measure the impact on the changes on CTA4 (less prominent login button). Regardless, I think the change makes a lot of sense and we should see its indirect impact on the resulting account creations.
Recap: 1/ Update CTA4: - get rid of "or" and "maybe later" - turn "log in" into a text-link rather than a button 2/ Disable all clicktracking, but keep campaign=aftv5_cta4 in CTA4-to-UXAC link 3/ Change bucketing: - Anons: 100% to CTA4 - Editors: 90% to CTA5, 10% to CTA6 I've pushed the above to Gerrit: https://gerrit.wikimedia.org/r/#/c/30771/ It's on prototype as well, though due to some obscure error caused by having a too outdated install on there, you probably won't get to the CTA. I'll attache a screenshot of what CTA4 will now look like.
Created attachment 11281 [details] CTA4 after the changes
Thanks Matthias, waiting for Fabrice to confirm if he's ok with the CTA setup.
Thanks, Matthias and Dario, this looks good to me! Only one minor nitpick: after seeing the screenshot, it seems like the word 'or' fulfilled a useful function after all. So if it's easy, you might insert it back in small gray text between 'Create an account' and 'Login', with a couple extra spaces on either side to keep them separated. Thanks for the rapid turnaround!
