Last modified: 2012-08-04 21:10:45 UTC
Let's test that we are collecting the data we need through clicktracking: * Edit CTA clicks * Edit CTA impressions * Learn more CTA clicks * Learn more CTA impressions * Edit saves * Edit attempts * Feedback link clicks (Option C, vertical button) This is based on this requirement: * Use the ClickTracking extension (http://www.mediawiki.org/wiki/Extension:ClickTracking) to measure click-through rate of CTA (clicks/CTA impressions) and edit save rate (successful edits/attempted edits).
Here are the action items we discussed towards this goal: 1) Reha will look at the clicktracking DB on prototype (with help from Roan if needed) to see if all the above data are being tracked as intended 2) Reha will add any above data items that are not being tracked 3) Reha will let us know by email what items are being tracked and what their labels are, as well as post this info on this ticket.
Here are some additional action items that will be needed on this issue: 4) Dario will then check the the labels and data, and make sure that they are usable for data/metrics analysis 5) We may need to separate the tables so that they do not interfere with AFTv4 data collection (Dario, Roan and Reha to figure out offline) 6) The DB label for this clicktracking data also needs to be added to the AFTv5 tables See data/metrics plan for more info on these issues. http://meta.wikimedia.org/wiki/Research:Article_feedback/Data_and_metrics
This is the complete list of events/UI elements for which we need to collect clicktracking data. Note that this only covers code released as of the 1.0 launch. Please let me know off list if you have any questions, if the list is ok I will start adding labels for each of these events. ==Feedback button== * feedback button clicks ==AFT5 Option 1 (Yes/No), bottom == * help button clicks * terms of use link clicks * yes button clicks * no button clicks * submit button clicks ==AFT5 Option 1 (Yes/No), overlay == * impressions (overlay only) * help button clicks * disable button clicks (overlay only) * terms of use link clicks * yes button clicks * no button clicks * submit button clicks ==AFT5 Option 2 (4 tabs), bottom== * help button clicks * terms of use link clicks * tab-1 clicks * tab-2 clicks * tab-3 clicks * tab-4 clicks * submit button clicks ==AFT5 Option 2 (4 tabs), overlay== * impressions (overlay only) * help button clicks * disable button clicks (overlay only) * terms of use link clicks * tab-1 clicks * tab-2 clicks * tab-3 clicks * tab-4 clicks * submit button clicks ==AFT5 Option 3 (Rating), bottom== * help button * terms of use link * rating clicks (1 event per rating, I don't care about individual stars) * submit button ==AFT5 Option 3 (Rating), overlay == * impressions (overlay only) * help button * disable button (overlay only) * terms of use link * rating clicks (1 event per rating, I don't care about individual stars) * submit button ==CTA-1 (Edit), bottom== * impressions * help button * learn how to edit link clicks * Edit this page button clicks * Edit save attempts * Edit save completed * feedback saved backlink clicks ==CTA-1 (Edit), overlay== * impressions * help button * disable button (overlay only) * learn how to edit link clicks * edit this page button clicks * edit save attempts * edit save completed * feedback saved backlink clicks ==CTA-2 (Learn more), bottom== * impressions * help button * learn how to edit button clicks * feedback saved backlink clicks ==CTA-2 (Learn more), bottom== * impressions * help button * disable button (overlay only) * learn how to edit button clicks * feedback saved backlink clicks
Thanks, Dario, this is very thorough! Yoni or Reha, do these requests work for you? Anything in there seem like a show-stopper for 1.0 launch? I only have a couple comments on this list, which looks pretty comprehensive to me: 1) Should we (can we) track clicks in the text area of the forms? 2) CTA2's button is 'Learn more' -- not 'Learn to Edit' -- and we may not need to track the feedback-saved backlink clicks for it, since it doesn't take you to the edit page, just to the tutorial 3) Dario, when preparing label names for OmniTI to use for this, I recommend using words that are very close to what the buttons or UI actually say, for clarity (e.g. 'Post' instead of 'Submit', etc.) Other than that, I think you have covered the key data we are looking to track, thanks!
Fabrice, here's a few comments 1) nice to have, definitely not a show-stopper 2) that's correct, it's confusing that we have a different wording for CTA1 and CTA2, but I'll stick to whatever language we use. We do need to track feedback-saved backlink clicks as they are one of the possible exit points from the CTA. Just to clarify I am referring to the link that appears in the "Your feedback has been saved" – I am not sure what that currently links to. 3) agreed
the last heading in the above list should actually read: ==CTA-2 (Learn more), overlay ==
Here's a shorter, high-priority list of events if the above list is too long to be implemented before the 1.0 deployment: ==Feedback button== * feedback button clicks ==AFT5 Option 1 (Yes/No), bottom == * submit button clicks ==AFT5 Option 1 (Yes/No), overlay == * impressions (overlay only) * disable button clicks (overlay only) * submit button clicks ==AFT5 Option 2 (4 tabs), bottom== * submit button clicks ==AFT5 Option 2 (4 tabs), overlay== * impressions (overlay only) * disable button clicks (overlay only) * submit button clicks ==AFT5 Option 3 (Rating), bottom== * submit button ==AFT5 Option 3 (Rating), overlay == * impressions (overlay only) * disable button (overlay only) * submit button ==CTA-1 (Edit), bottom== * impressions * edit this page button clicks * edit save attempts * edit save completed ==CTA-1 (Edit), overlay== * impressions * disable button (overlay only) * edit this page button clicks * edit save attempts * edit save completed ==CTA-2 (Learn more), bottom== * impressions * learn how to edit button clicks ==CTA-2 (Learn more), overlay== * impressions * disable button (overlay only) * learn how to edit button clicks
The short list is nearly done (as of r106251) Here are the tracking ids I've selected (excluding the prefix to designate it for this version of AFTv5). I'm operating under the assumption that because you listed the overlay and bottom states separately, you wanted them tracked separately. Please let me know if I should make any of these share a tracking id. ==Feedback button== * feedback button clicks - 'vertical-link' -- also 'toolbox-link' for clicks on the toolbar link (this was already being tracked) ==AFT5 Option 1 (Yes/No), bottom == * submit button clicks - 'option1-submit' ==AFT5 Option 1 (Yes/No), overlay == * impressions (overlay only) - 'option1-impression-overlay' * disable button clicks (overlay only) - 'option1-overlay-close' * submit button clicks - 'option1-submit-overlay' ==AFT5 Option 2 (4 tabs), bottom== * submit button clicks - 'option2-submit-bottom' ==AFT5 Option 2 (4 tabs), overlay== * impressions (overlay only) - 'option2-impression-overlay' * disable button clicks (overlay only) - 'option2-overlay-close' * submit button clicks - 'option2-submit' ==AFT5 Option 3 (Rating), bottom== * submit button - 'option3-submit-bottom' ==AFT5 Option 3 (Rating), overlay == * impressions (overlay only) - 'option3-impression-overlay' * disable button (overlay only) - 'option3-overlay-close' * submit button - 'option3-submit-overlay' ==CTA-1 (Edit), bottom== * impressions - 'cta-edit-impression-bottom' * edit this page button clicks - 'cta-edit-button-click-bottom' * edit save attempts - NOT YET WORKING * edit save completed - NOT YET WORKING ==CTA-1 (Edit), overlay== * impressions - 'cta-edit-overlay-impression' * disable button (overlay only) - 'cta-edit-overlay-close' * edit this page button clicks- 'cta-edit-button-click-overlay' * edit save attempts - NOT YET WORKING * edit save completed - NOT YET WORKING ==CTA-2 (Learn more), bottom== * impressions - 'cta-learn-more-bottom-impression' * learn how to edit button clicks - 'cta-learn-more-button-click-bottom' ==CTA-2 (Learn more), overlay== * impressions - 'cta-learn-more-overlay-impression' * disable button (overlay only) - 'cta-learn-more-overlay-close' * learn how to edit button clicks- 'cta-learn-more-button-click-overlay'
Tweaks to naming in r106267: Added "optionX-" to the front of all the cta events, and to the feedback links General style is now [option identifier]-([cta identifier]-)[event]-[bottom/overlay]
Copying here for the record some important tweaks that Reha and I just discussed over IRC: * vertical-link events should be prefixed with the option identifier * all CTA events should be prefixed with the option and the placement identifier The general naming scheme we'll be using is the following: 'ext.articleFeedbackv5@' + version + '-' + key with version as the version number set in the configuration and key the event identifier: * [option id]-[event]-[placement id] * [option id]-[cta id]-[event]-[placement id]
This should be fixed now. Complete list of tracking ids: option1-submit-overlay option2-submit-overlay option3-submit-overlay option1-submit-bottom option2-submit-bottom option3-submit-bottom option1-impression-overlay option2-impression-overlay option3-impression-overlay option1-cta_edit-impression-overlay option2-cta_edit-impression-overlay option3-cta_edit-impression-overlay option1-cta_learn_more-impression-overlay option2-cta_learn_more-impression-overlay option3-cta_learn_more-impression-overlay option1-cta_edit-impression-bottom option2-cta_edit-impression-bottom option3-cta_edit-impression-bottom option1-cta_learn_more-impression-bottom option2-cta_learn_more-impression-bottom option3-cta_learn_more-impression-bottom option1-close-overlay option2-close-overlay option3-close-overlay option1-cta_edit-close-overlay option2-cta_edit-close-overlay option3-cta_edit-close-overlay option1-cta_learn_more-close-overlay option2-cta_learn_more-close-overlay option3-cta_learn_more-close-overlay option1-cta_learn_more-button_click-overlay option2-cta_learn_more-button_click-overlay option3-cta_learn_more-button_click-overlay option1-cta_learn_more-button_click-bottom option2-cta_learn_more-button_click-bottom option3-cta_learn_more-button_click-bottom option1-cta_edit-button_click-overlay option2-cta_edit-button_click-overlay option3-cta_edit-button_click-overlay option1-cta_edit-button_click-bottom option2-cta_edit-button_click-bottom option3-cta_edit-button_click-bottom option1-cta_edit-edit_attempt-overlay option2-cta_edit-edit_attempt-overlay option3-cta_edit-edit_attempt-overlay option1-cta_edit-edit_attempt-bottom option2-cta_edit-edit_attempt-bottom option3-cta_edit-edit_attempt-bottom option1-cta_edit-edit_success-overlay option2-cta_edit-edit_success-overlay option3-cta_edit-edit_success-overlay option1-cta_edit-edit_success-bottom option2-cta_edit-edit_success-bottom option3-cta_edit-edit_success-bottom *NB: Resolved in code; not yet pushed.
Added: option1-impression-bottom option2-impression-bottom option3-impression-bottom Added at random 1% sampling: option1-init option2-init option3-init Both in r108063.
It appears that the following events are not being captured at all in clicktracking.log, unless their name is different from the one specified here and at: http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Full_list_of_events option1-cta_edit-button_click-overlay option2-cta_edit-button_click-overlay option3-cta_edit-button_click-overlay option1-cta_edit-button_click-bottom option2-cta_edit-button_click-bottom option3-cta_edit-button_click-bottom Counting these events is critical to measure the click-through and completion rate for the edit CTA.
Found and fixed this bug in r111179; tested locally and on prototype. I noticed while I was working on it that although the Learn More CTA's button clicks are tracked, the Edit CTA's learn how to edit link is not. Dario's draft list (comment #3) includes it, but his priority revision (comment #7) does not. Is this correct? Are we ready to start tracking it now.
Thanks for the fix, Reha. "Learn how to edit" is not as critical as the edit button clicks, but it would be nice to have if it's simple to enable it.
Merged to both 1.18wmf1 and 1.19wmf1 in r111506 Pushed to site in 1.18wmf1 config
Added clicktracking requirements for phase 2 http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Phase_2_.28placement.29 ==New bucket definitions:== * option1A: trigger A + widget in the overlay (bottom widget is still displayed) * option1E: trigger E + widget in the overlay (bottom widget is still displayed) * option1X: no trigger + widget at the bottom of the page only ==List of events== option1A-init (1%) option1E-init (1%) option1X-init (1%) option1A-impression-overlay option1E-impression-overlay option1A-impression-bottom option1E-impression-bottom option1X-impression-bottom option1A-submit-overlay option1E-submit-overlay option1A-submit-bottom option1E-submit-bottom option1X-submit-bottom option1A-cta_edit-impression-overlay option1E-cta_edit-impression-overlay option1A-cta_learn_more-impression-overlay option1E-cta_learn_more-impression-overlay option1A-cta_edit-impression-bottom option1E-cta_edit-impression-bottom option1X-cta_edit-impression-bottom option1A-cta_learn_more-impression-bottom option1E-cta_learn_more-impression-bottom option1X-cta_learn_more-impression-bottom option1A-close-overlay option1E-close-overlay option1A-cta_edit-close-overlay option1E-cta_edit-close-overlay option1A-cta_learn_more-close-overlay option1E-cta_learn_more-close-overlay option1A-cta_learn_more-button_click-overlay option1E-cta_learn_more-button_click-overlay option1A-cta_learn_more-button_click-bottom option1E-cta_learn_more-button_click-bottom option1X-cta_learn_more-button_click-bottom option1A-cta_edit-button_click-overlay option1E-cta_edit-button_click-overlay option1A-cta_edit-button_click-bottom option1E-cta_edit-button_click-bottom option1X-cta_edit-button_click-bottom option1A-cta_edit-edit_attempt-overlay option1E-cta_edit-edit_attempt-overlay option1A-cta_edit-edit_attempt-bottom option1E-cta_edit-edit_attempt-bottom option1X-cta_edit-edit_attempt-bottom option1A-cta_edit-edit_success-overlay option1E-cta_edit-edit_success-overlay option1A-cta_edit-edit_success-bottom option1E-cta_edit-edit_success-bottom option1X-cta_edit-edit_success-bottom option1A-triggerA-click-overlay option1E-triggerE-click-overlay option1A-edit_tab_link-click option1E-edit_tab_link-click option1X-edit_tab_link-click option1A-section_edit_link-click option1E-section_edit_link-click option1X-section_edit_link-click option1A-edit_tab_link-edit_attempt option1E-edit_tab_link-edit_attempt option1X-edit_tab_link-edit_attempt option1A-section_edit_link-edit_attempt option1E-section_edit_link-edit_attempt option1X-section_edit_link-edit_attempt option1A-edit_tab_link-edit_success option1E-edit_tab_link-edit_success option1X-edit_tab_link-edit_success option1A-section_edit_link-edit_success option1E-section_edit_link-edit_success option1X-section_edit_link-edit_success All events used in Phase 1 should be disabled
reopening by Derio's request. Reha: implement the rest of the requirements.
Added the following six events to the specs for Phase 2: /* clicks on the x button */ option1A-disable_button_click option1E-disable_button_click /* impressions of the flyover */ option1A-disable_flyover-impression option1E-disable_flyover-impression /* clicks on the go to preferences button from the flyover */ option1A-disable_gotoprefs_click option1E-disable_gotoprefs_click Unless it's trivial, I don't think we need to count successfully saved changes in user preferences: logging the above events should be sufficient to estimate how many registered users have the intention to disable AFT. I updated the specs on Meta accordingly http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Phase_2_.28placement.29
Phase 2 clicktracking (including the disable flyover) checked in -- r114189
Phase 3 clicktracking specs added: http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking#Stage_3_.28Impact_on_engagement.29 I prepared these specs based on a number of (partly arbitary) assumptions on the implementation of the Option4 fallback for non-editable pages. All these assumptions will need to be reviewed before we start adding the tracking codes.
Phase 3 clicktracking committed and submitted to gerrit for review: https://gerrit.wikimedia.org/r/5141 https://gerrit.wikimedia.org/r/5142
Discussed this with Dario -- it makes more sense to keep clicktracking in just one place and only open a bug if something's gone wrong. From here on out, clicktracking *requirements* go here: http://meta.wikimedia.org/wiki/Research:Article_feedback/Clicktracking Closing.
