['\n', ':: {{ping|Kww}}\n', ':: Hey Kww,\n', ":: That's a totally legitimate question. These are some of the main types of issues we're dealing with:\n", '::* Category A: straightforward bugs, \'\'e.g.\'\' "When I click this button in the dialog, it stops responding".\n', '::* Category B: performance issues, \'\'e.g.\'\' "It takes 25 seconds to save the page".\n', '::* Category C: user experience issues, \'\'e.g.\'\' "The reference dialog is not intuitive for me".\n', '::* Category D: missing functionality, \'\'e.g.\'\' "I can\'t insert or edit maths equations".\n', ";: Our ability to address issues in ''each'' category is greatly enhanced by a significantly sized pool of users making real world edits using VisualEditor, and greatly decreased by seeing that set of users drop to a trickle, or a highly self-selective pool.\n", ":: However, as I noted, you are correct that we don't require the firehose of usage that we're seeing today. That firehose has the benefit that it's truly unbiased -- IPs, new users, and long-time Wikipedians have all used VisualEditor in the last few weeks. But we understand that this is disruptive, and costing us goodwill. Sorry. :-( However, a compromise like the one we're suggesting above would help maintain a reasonable level of usage, while more fully informing users that they're participating in a beta test, and not changing the default experience for anyone.\n", ";: Let me try to explain why each category of issue benefits from a large ''and'' diverse pool of users making real world edits.\n", "::;Category A issues: Bugs of this type often depend on VisualEditor being in a certain state (''e.g.'' a certain element on the page being selected), or only occur sporadically, or for users with certain configurations. We may be able to isolate and think we've fixed the bug under testing conditions, but it continues to occur in real-world conditions. Were we to go to 0 users and do all our testing internally, we'd do months of development and fix a lot of these issues, release again and have to re-open many of the bugs, because it turns out that users still encounter them under certain conditions. Users yell at us because the software still has bugs. That's fairly likely, and doesn't serve anyone well. \n", "::: In order to catch issues and regressions while we're developing code, we use unit tests, automated browser tests, and automated parser round-tripping tests. We have a continuous integration pipeline that lets us know when a commit breaks a given test. All aspects of the testing pipeline can be improved, of course, but even with a solid testing pipeline, you'll run up against the limitations of automated testing. In particular, maintaining more than a basic set of cross-browser tests requires very large development effort in its own right, because browser automation tends to be brittle as the user interface changes.\n", "::: Manual testing (literally paying people to follow test plans, perform exploratory testing, ''etc.'') also only gets us so far. If you've done Web development, you will know, for example, that the number of browser variants to support is ever-increasing, with browser makers shifting to very short release cycles, and each release exhibiting slightly different characteristics. As you go down in versions one-by-one from Firefox 22, you might literally see the same bug manifest in a dozen different ways.\n", "::: So, in order to really track an issue under real-world conditions and ensure that it's actually fixed, and in order to be complement our own testing efforts with the radically heterogeneous environments of our users, it is helpful to have a continued high level of real-world usage as development continues.\n", "::; Category B issues: Performance may seem like the most straightforward thing to optimise without having a significantly-sized user base, but it's actually not. The reason is that such issues often require highly-unusual optimisation strategies. The largest area for performance improvements isn't in the server-side code, but in the client-side code that manipulates the document object model. We're highly likely to encounter new issues ''as'' we optimise performance, and we do need new reports as we iterate, rather than by the time we reach a final state.\n", '::; Category C and D issues: Essentially, any time we make a change to the user experience, we need to see whether it works. Not just "works for us at WMF", or even "works for our projection of what real-world users would expect", but actually works. The diffs of real users making edits using VisualEditor are essential for this. Does the number of references being added go up? Does the number of unintended <code><nowiki><nowiki></nowiki></code>s go down? Are users successfully using the table editor? Once again, doing this in a "waterfall" model where we come to you in 6 months with a shiny table editor is the best way to ensure that you \'\'won\'t\'\' be satisfied with the changes we make, and that we won\'t either. Aiming, always, for the [[minimum viable product]] helps us iterate under real world conditions.\n', ":: In sum, once again, while I completely understand the frustration with the opt-out beta period, we do need to build an unbiased, reasonably representative set of VE users in order to effectively support addressing any kind of issue you encounter, rather than quietly slaving away for a long time and then releasing a product that doesn't meet user needs and has lots of new and old bugs that only surface in real-world conditions.\n", ":: Hope that's reasonable. [[User:Jdforrester (WMF)|Jdforrester (WMF)]] ([[User talk:Jdforrester (WMF)|talk]]) 03:17, 1 August 2013 (UTC)\n"]