⚓ T314534 Share Edit Notices feedback to Editing team
Page MenuHomePhabricator

Share Edit Notices feedback to Editing team
Closed, ResolvedPublic

Description

Background
The Editing reached out to ask:

do y'all have any data about how introducing edit notices within the android app impacted editing behavior? e.g. changes in the rates at which people complete the edits they start and the rates at which the edits people make are reverted.

The team will use the data to make decisions about Edit Notices on Web. This is a somewhat time sensitive request.

The Task
Due to our team not instrumenting Edit Notices we will instead share the following indicators:

  • changes in edit completion rate
  • changes in the rate at which edits are reverted

Note* Edit Notices became live in the app at the end of November 2021.

Related Objects

Event Timeline

JTannerWMF renamed this task from [placeholder] Edit Notices feedback to Editing team to Share Edit Notices feedback to Editing team.Aug 8 2022, 3:15 PM
JTannerWMF reassigned this task from JTannerWMF to SNowick_WMF.
JTannerWMF triaged this task as Medium priority.
JTannerWMF updated the task description. (Show Details)
JTannerWMF added subscribers: ppelberg, MNeisler, JTannerWMF.

@ppelberg and @MNeisler
I've done analysis for both iOS and Android post-notification editing data, can share data but most useful right away would be the iOS editing impact analysis I did in June 2022: https://phabricator.wikimedia.org/T308793#8018661, links to all data are in the ticket, please let me know if you have trouble accessing. Note that data is not edit notice specific, just notifications in general so may not be useful.

We also have Android analysis data that I can share but would benefit from a summary which I will put together, wanted to get you the most comprehensive reporting first. Apologize for the delay.

Also relevant - Android editor analysis - notifications by type: Data

Last Android analysis: results of our ABC test for a new notification Icon, in this deck are slides that show notification interactions w/edit counts before and after notifications release. Results are segmented by our test groups. Deck

Note: This analysis uses our app internal editing event metrics so these datasets do not include revert rates. Using test groups that require by Country segmentation precludes us from using mediawiki_history edit data that includes reverts.

Hopefully this more clearly answers what your team was looking for w/r/t edit notice impact, I pulled edit counts and revert rates from before and after we implemented edit notices. The following is monthly edit data (edit notices were released in 2021-11). It appears that edit notices did not discourage edit completion, edit revert rates also decreased.

MonthAndroid EditsRevertsRevert Rate
2021-10136563117228.6%
2021-11141874118068.3%
2021-12152971119817.8%
2022-01159345126067.9%
2022-02160626120317.5%

Hopefully this more clearly answers what your team was looking for w/r/t edit notice impact, I pulled edit counts and revert rates from before and after we implemented edit notices. The following is monthly edit data (edit notices were released in 2021-11).

This is helpful – thank you for pulling this together, @SNowick_WMF! A quick clarifying question for you...

It appears that edit notices did not discourage edit completion...

Would it be accurate for me to think that you arrived at the conclusion above by observing that the total number of Android edits increased following the deployment of edit notices in 2021-11?