103 KiB
Managing Episodic Volunteers in Free/Libre/Open Source Software Communities
Ann Barcomb, Klaas-Jan Stol, Brian Fitzgerald, and Dirk Riehle
Abstract—We draw on the concept of episodic volunteering (EV) from the general volunteering literature to identify practices for managing EV in free/libre/open source software (FLOSS) communities. Infrequent but ongoing participation is widespread, but the practices that community managers are using to manage EV, and their concerns about EV, have not been previously documented. We conducted a policy Delphi study involving 24 FLOSS community managers from 22 different communities. Our panel identified 16 concerns related to managing EV in FLOSS, which we ranked by prevalence. We also describe 65 practices for managing EV in FLOSS. Almost three-quarters of these practices are used by at least three community managers. We report these practices using a systematic presentation that includes context, relationships between practices, and concerns that they address. These findings provide a coherent framework that can help FLOSS community managers to better manage episodic contributors.
Index Terms—Best practices, community management, episodic volunteering, free software, open source software
1 INTRODUCTION
Free/Libre/Open Source Software (FLOSS) research has traditionally divided contributors into core and periphery, where core describes the minority of top developers who contribute 80 percent of the code and the periphery describes all other developers [1], [2], [3]. This focus on the volume of contributions assumes a homogenized periphery, without any further distinction within that group. Further, by its very definition this distinction has an exclusive focus on code contributions, ignoring the many other types of contributions that are made to FLOSS projects. To better understand the periphery of FLOSS communities, several researchers have begun to differentiate participants within the periphery, based on the frequency and duration of their participation [4], [5], [6], [7]. In earlier work, we have drawn upon the concept of episodic volunteering (EV) from the volunteering literature to describe the subset of peripheral contributors whose contributions are short-term or infrequent [8], [9], in contrast to habitual contributors, whose contributions are “continuous or successive” [10]. In so doing, we have also reconsidered the definition of contribution, expanding it from software (or code) contribution to any type of activity within a FLOSS community [6]. By using this alternative lens on FLOSS communities, we found evidence for a wide range of contributions that episodic volunteers have made [6]. Based on a qualitative survey of 13 FLOSS communities, we developed a detailed understanding from the perspectives of both episodic volunteers and community managers. Based on this, we established an initial set of recommendations to engage episodic volunteers. A key concern in the context of episodic volunteering is whether these volunteers return to make further contributions. Drawing on the general volunteering literature, we evaluated a theoretical model that helps explain retention of episodic volunteers.
In this article we extend this line of research on EV in FLOSS communities. Episodic contributors represent a class of participants that can make a wide range of valuable contributions to FLOSS projects [6]. By their very nature, their participating behavior is incidental and not continuous, and so it is of particular interest to understand how episodic contributors can be “retained,” which in this context refers to them returning to a project to contribute again, rather than converting them into habitual contributors. Retention is appealing because returning contributors require less assistance than newcomers [11] and retention is one of the key factors in FLOSS project sustainability [12], [13], [14], [15], [16]. However, evidence from the general volunteering literature suggests that many organizations do not have clear strategies in place to effectively manage episodic contributors [11], [17]. Organizations may also face internal resistance in implementing these changes, as episodic contributors may be negatively perceived as costing more in resources than they deliver in contributions [18].
Despite these challenges, EV is an increasingly important topic in volunteer management due to the increase in and preference for this kind of work [8], [19], [20], [21], [22]. Adapting to the changing volunteering context is necessary for the sustainability of non-profit organizations [22]. FLOSS it has long been observed that many contributors are episodic, for instance in the case of bug reporting [2], [6], [23], [24], [25]. Furthermore, a number of benefits have been attributed to peripheral contributors—increased identification of legal issues such as copyright infringement, and high-quality bug fixes, for example [14], [26]. Hence, given the increased recognition of the importance of episodic volunteers and their contributions, it is imperative to study how to manage episodic volunteers in FLOSS communities.
A major change in FLOSS communities over the last decade has been the increase in firms’ involvement in open source development although volunteers remain important participants [27], [28], [29]. Many companies in different sectors use software which is developed by external FLOSS projects [30], and consequently many firms now employ developers to contribute to specific open source projects that they identify as critical to their business. Paid development does not negate the need to understand episodic participation. Even in company-dominated FLOSS communities, external developers still contribute a significant proportion of commits [31]. Additionally, from the perspective of the community, paid developers employed by external firms cannot be directed as employees [32], [33]. Although there are differences between paid contributors and other participants [28], paid contributors’ participation is sometimes episodic from the perspective of the community. Our research considers episodic participation from the community perspective, and consequently we adopt the broadest definition of volunteering, to encompass anyone engaging in FLOSS contributions who is not directly sponsored by the FLOSS community [6]. This broad definition allows us to identify practices which can actually be used by communities, without any concern for whether or not contributors are paid or sponsored by a firm. When paid contributors affect community managers’ concerns and practices, this is explicitly noted in our findings.
FLOSS research has been challenged for its reliance on studying forms of participation which can be readily observed through data mining, notably code contributions, bug reports, and mailing lists [34], [35]. Exclusion of non-code contributors limits the applicability of research on larger FLOSS communities, which depend not only on code contributions but also a wide range of other activities, such as planning, advocacy, mentoring, and event organization [35], [36], [37], [38]. Both unpaid and paid contributors can participate in a range of activities within FLOSS communities [39].
Despite extensive research on community practices, e.g., [3], only two studies have focused specifically on episodic participation, and neither focused on identifying an extensive list of practices [6], [40]. The fact that specific practices have been proposed for other peripheral sub-groups, namely newcomers [41], [42], suggests that FLOSS communities may be using different practices, or adapting existing practices to different ends, in order to manage episodic contributors. Hence, our study had the following objectives:
- Identify the concerns community managers have about episodic volunteers.
- Identify the practices that community managers are using, or envisage using, to address their concerns about episodic volunteers.
To address these objectives, we conducted a Delphi study, which is a structured communication technique involving a panel of experts. We drew on the experience of FLOSS community managers to identify the concerns community managers have with EV, the practices they use—or consider using—to manage EV, and preliminary suggestions for how practices could be combined. This article makes the following contributions toward understanding the management of EV in FLOSS:
- A prioritized list of 16 EV community manager concerns;
- An extensive collection of practices which might be used to manage EV (74 percent are being used by at least three community managers), which includes connections to the concerns previously identified, as well as relationships between practices;
- Workflows proposed by community managers which demonstrate how practices can be combined.
The remainder of the article is organized as follows. Section 2 reviews previous work that investigated open source communities, volunteers, and in particular the role of episodic contributors. Section 3 presents the Delphi research approach that we adopted, including a discussion of participant selection, data collection, and data analysis procedures. Section 4 presents the findings of the study by presenting a set of practices and concerns. Section 5 concludes by discussing our findings, the limitations of the study, and an outlook to future work.
2 RELATED WORK
This section reviews prior work on peripheral contributors and episodic volunteering in FLOSS communities.
2.1 Peripheral Contributors in FLOSS Communities
One of the earliest conceptions of the structure in FLOSS communities is the so-called Onion model [1], [43]. The Onion model depicts increasing numbers and decreasing engagement moving from the innermost core to the outermost passive users. The core contains the most prolific developers, often described as the people who create 80 percent of the code [2]. Beyond the core is the periphery, who contribute fewer lines of code.
Although much of the earlier research focused on the core (e.g., [2], [24]), there is now significant understanding of both the importance of the periphery and the motivations of peripheral participants. Peripheral contributors provide a range of benefits:
- Bringing new knowledge to the project [26], [44], [45], [46];
- Raising awareness of the project [46], [47], [48];
- Providing new potential core contributors [26], [45], [49], [50], [51];
- Proposing new features [44], [52];
- Contributing new code [26], [44], [45], [53];
- Finding and reporting bugs [54];
- Ensuring members’ behavior abides by community norms [26].
FLOSS developer motivations have been extensively studied. Motives are usually characterized as intrinsic motives, inherent to the job, such as altruism and enjoyment, internalized extrinsic motives such as reputation and reciprocity, and extrinsic motives such as career and salary [55]. Peripheral contributors tend to have the same set of motivations as core developers [37], but those with extrinsic motives are less likely to continue to participate [45], [56]. In particular, peripheral contributors are more likely to seek out opportunities which afford them greater recognition with stakeholders and the chance to gain reputation [45]. Extrinsic motives, such as the desire to build a reputation and gain recognition, are more widespread among peripheral developers than core developers [45].
Recent work has begun to study the periphery more closely to identify and distinguish different types of contributors. One dimension often used to distinguish is the frequency of participation. Groups that are distinguished by the frequency of their participation are newcomers [41], [57], [58], [59], [59], [60], [61], [62], people who attempt to become contributors [63], and one-time contributors [5], [40], [56]. In earlier work, we have linked the general episodic volunteering literature to the periphery [6]. The disaggregation of the periphery by frequency of contribution could also be viewed as an extension to, rather than a departure from the Onion model. The outer layers—active users and passive users—are already defined by their own actions irrespective of the contributions of others. Active users engage with the project, for instance by supplying bug reports, while passive users only use the software. Disentangling the homogenized periphery into sub-categories distinguished by frequency of participation refines the Onion model and allows for the identification of distinct attributes of different groups within the periphery.
In the Onion model, the different layers describe how people contribute to the software, whereas FLOSS projects include many other ways to get involved [35], [36], [37]. Carillo and Bernard [64] described code-centricity as a limitation:
“By stereotyping FOSS projects as communities of developers loosely collaborating on a FOSS-licensed software project via an online project platform, we disregard the massive amount of information that is not captured on platforms and also neglect the myriad of non-code related tasks and roles without which a project could not be what it is.”
Emphasis on code contributions within FLOSS communities may not only devalue other types of contributions, but may specifically disadvantage women [65]. Other studies have found that women’s participation in FLOSS remains low in both code and non-code activities, including leadership [66], [67], [68]. Nafus’s [65] participant observation study of FLOSS contributors found that “men monopolize code authorship and simultaneously de-legitimize the kinds of social ties necessary to build mechanisms for women’s inclusion.” Research has also demonstrated that some barriers to entry for newcomers are gendered [60], [69], and that gender may influence retention among episodic contributors [7]. Because code contributors do not represent the entire community in terms of the diversity of work, and may additionally be demographically unrepresentative, we argue the importance of including non-code contributions in our study. This emphasis makes the EV concept, which originates in the general volunteering literature rather than the software engineering literature, an appropriate lens for the study because it places no particular emphasis on any one type of contribution.
2.2 Episodic Volunteering
Episodic volunteering is a term from the general volunteering literature describing short-term or infrequent participation. Although a particular engagement may be of limited duration, retention of episodic contributors is possible. In the context of EV, retention does not mean conversion to habitual participation but repeated engagement with the same organization. In a systematic review of the EV literature, Hyde et al. [70] identified retention as a key topic in need of further research. Retention remains a compelling subject because returning volunteers require less training [11] and retention is one measure of stability in FLOSS [13], [14], [15]. The general volunteering literature on the retention of episodic contributors has largely focused on explaining the factors that lead to retention, such as satisfaction with the previous volunteering experience, intention to return, and availability [10], [71], [72]. In the FLOSS domain, Steinmacher et al. [73] found that higher quality email responses encouraged retention among newcomers. Meanwhile, Labuschagne and Holmes [57] critically examined Mozilla’s onboarding programs and found that it may not result in long-term contributors, despite the fact that mentored newcomers consider the program valuable. A study evaluating five potential EV retention factors found that satisfaction, community commitment, and social norms correlate with intention to remain [7].
Another important problem in general volunteering is how organizations incorporate EV [17]. Although EV is sometimes viewed as disruptive, it is widespread and a reality that requires organizations to reconsider their strategies [18], [19], [45], [74]. Volunteer agencies can adjust to the expectations of episodic contributors by offering more flexibility in commitment, reducing training requirements, increasing the social element of service, and recognizing volunteers [75]. Volunteer coordinators can also identify tasks that are suitable for episodic contributors, which may include one-off contributions at events and on-going but non-specialized work [11]. Evaluation of suitable tasks can be done systematically by applying a ‘volunteer scenario’ approach that categorizes volunteer assets, volunteer availability, and potential assignments [76].
While there is no single work that has collected a comprehensive set of practices for managing EV in FLOSS, previous studies have proposed practices for managing FLOSS contributors. Previously, we identified 20 potential practices for EV management by evaluating existing FLOSS practices in light of factors associated with the retention of episodic contributors and prior general volunteering recommendations [6]. Meanwhile, Steinmacher et al. [41] identified nine practices for communities onboarding new contributors and corresponding recommendations for new contributors. We consider practices for newcomers relevant to the study of EV because community managers cannot distinguish the future episodic volunteer from the future habitual volunteer [72] when they make their first contribution. This study updates this line of work by drawing on the expertise of community managers. At the time of our first study [6], we found very limited evidence of community managers managing EV. This approach increases the scope and number of practices identified. First, we examine both practices which are already being used to manage EV as well as practices that experts think might be appropriate, and distinguish between speculation and observed practice. Second, we look at most of the volunteer process, from onboarding to retention, excluding only recruitment.
3 Study Design
In this section we outline the Delphi research method, elaborate the participant selection, and data collection and analysis methods.
3.1 Research Method
Our research is concerned with understanding current practices for managing episodic contributors, and also proposes practices that may be helpful for managing EV. The Delphi method was developed as a way of finding the collected opinions of a group of experts and works on the assumption that multiple experts are better able to arrive at more accurate solutions to problems. Anonymity between participants is used to prevent participants with high status or reputation from having a disproportionate influence [77], [78], [79]. The Delphi approach is suitable for complex problems [80], when solutions do not yet exist and may be best explored through the subjective judgments of an informed group of experts [77], [81].
While not common in software engineering research, the Delphi method has previously been used to study complex topics such as tailoring of agile methods [82] and the adoption of tools by FLOSS developers [83]. Delphi studies typically comprise several rounds of data collection—as participants are exposed to new information in every round, they may develop new insights through iteration and exposure to others’ ideas. The Delphi method can also be conducted asynchronously, which was of particular importance in our context given the geographic distribution of open source experts.
The traditional Delphi method focuses on achieving consensus. As it has evolved, a variant known as the policy Delphi has emerged. A policy Delphi study is appropriate when the purpose of the study is not to establish consensus but to identify the main arguments and positions [77]. We decided that a policy Delphi study rather than a traditional Delphi study would be more appropriate in our context, because we recognized that communities may have different goals when managing EV which could be driven by community size, cultural context, or types of contribution being considered. We wanted to articulate these constraints in order to provide context for the practices, rather than assume that one approach would be effective for all communities and activities within communities. However, we were also interested in generalizing common practices and concerns, and used the collation of the different rounds of data collection to achieve consensus of opinions.
We codify the results of our research in the form of a collection of practices, in the appendix [84], which can be found on the Computer Society Digital Library at http://doi.ieeecomputersociety.org/10.1109/TSE.2020.2985093. This ensures that the fruits of our research work can be used by practitioners, a key goal of our research.
EV management includes all phases of the volunteer management process. We explicitly excluded recruitment practices from consideration in our study because many of those are not specific to episodic volunteering. This focus was necessary to limit the scope of the study, which otherwise could overwhelm the participants and diffuse their focus. Although onboarding is another area where we expect overlap between habitual and episodic management, we decided to retain this part of the process in order to compare our results to a recent study summarizing onboarding practices for newcomers [41].
3.2 Participant Selection
Participant selection is a key aspect of a successful Delphi study [85]. Participants must be selected with care, and not chosen simply on the basis of availability [86].
We sought to select a panel of 20 to 25 participants, to ensure sufficient diversity even if some participants would stop participating in the study. This is within the recommended range of 15–30 participants [87]. Potential participants were identified in one of three ways. First, some approached us directly following presentations at practitioner conferences. Second, we identified people among our contacts, and people who were recommended to us by contacts. From these two groups we approached a subset which met our selection requirements, which we describe below. Third, we evaluated gaps in our coverage and sent cold emails to people we identified through online searches. The selection of participants was based not only on their enthusiasm for participation or connection to us, but also on the degree of diversity along the three selection dimensions (discussed below), as well as our expectation that the participants would be able to provide relevant input. Additionally, although gender has not, to our knowledge, been directly linked to community management, our awareness that gender can affect FLOSS participation experiences [60] inspired us to deliberately recruit female participants. In total, one-third of our participants were female. Table 1 summarizes the participants by community and their participation in the different rounds of our study.
To gain the full benefit of multiple perspectives, participants of a Delphi study should be diverse rather than homogeneous [88]. We identified three dimensions relevant to our study along which we expected differences of opinion to arise: size of community, contribution type, and country. We discuss each in detail below.
3.2.1 Size of Community
A previous study investigating the current state of EV in FLOSS discovered that the tasks considered appropriate for episodic contributors vary by community size [6]. For example, in smaller communities, translation is an ad-hoc task well-suited to EV. Larger communities have more complicated rules when translating, and a full cognizance of those rules requires more habitual participation. Organization size is also a factor commonly considered in studies identifying best practices. For example, in their case study of best practices for volunteer organizations, Carvalho and Sampaio [89] considered the size of volunteer organizations in terms of the numbers of beneficiaries, paid employees, and volunteers. Because there are many different ways to operationalize community size—number of users, number of developers, size of core—and because size is more continuous than categorical, we did not categorize communities by size, but instead sought to include a number of communities of different size.
All communities represented by our panel experts have more than a handful of contributors. This is justified because extremely small communities tend not to be concerned with developing a volunteer management process or workflow. The communities represented are shown in Table 1. In total, 22 communities were represented, and four of these communities (Debian, Ubuntu, KDE, OpenStack) were represented twice. Detailed descriptions of each community are provided in the appendix [84], available in the online supplemental material.
3.2.2 Contributor Activities
Much of FLOSS research has been code-centric, but in large communities people work in a number of activities, such as translation and maintaining web services [35]. Our earlier study on EV in FLOSS found that while episodic contributors can engage in all activities, some areas are considered more suitable than others, depending on the community [6]. We expect that the perspective of community managers might be influenced by the activities they engage in. We used the classification system introduced by Rozas [38] to describe the Drupal community, because it contains the most comprehensive categorization of FLOSS activities.
3.2.3 Country
FLOSS communities are international, although North American and European countries are disproportionately over-represented [90]. Geographic boundaries can be eliminated, but cultural barriers may remain. For example, in 2002, Nakakoji et al. [1] explained that Japanese programmers were reluctant to directly communicate with GNU GCC core developers because they saw them as superior programmers and wanted to keep a “respectful distance.” One difficulty with identifying cultural diversity is increasing globalization, which has led to intercultural identities and identification with not only country of birth, but also residence [91], [92]. We therefore considered both the country of origin as well as of residence.
Our participants represented 23 countries, spanning all populated continents: Argentina, Australia, Brazil, Cyprus, Czech Republic, France, Germany, Hungary, India, Ireland, Italy, Japan, Kenya, Peru, Romania, Singapore, Spain, South Korea, Tunisia, Uganda, Ukraine, the United Kingdom, and the United States. The appendix provides details about participants’ countries of residence and origin [84], available in the online supplemental material.
3.3 Data Collection and Analysis
Data collection was initiated in January 2018 and concluded in October 2018. The study comprised three rounds, as shown in Fig. 1.
In the first round, participants were asked to think of any concerns they had about EV, and how they might address them. All participants were engaged in community management, which was a precondition for participating in the study. Our participants had experience with close to six categories on average, and all were involved in multiple types of contributions. Table 2 shows a paraphrased list of contribution types along with a count of how many participants were engaged in each activity. The appendix provides a detailed list of each participant’s contribution types [84], available in the online supplemental material.
Table 1: Study Participants by Community and Study Participation
| ID | Community | Rounds participated |
|---|---|---|
| CM1 | (Anonymous) | ✓ |
| CM2 | Apache, RDO | ✓ |
| CM3 | ChakraLinux | ✓ |
| CM4 | CHAOSS | ✓ |
| CM5 | Debian | ✓ |
| CM6 | Drupal | ✓ |
| CM7 | Fedora | ✓ |
| CM8 | Fedora | ✓ |
| CM9 | Joomla! | ✓ |
| CM10 | KDE, NextCloud | ✓ |
| CM11 | KDE, Kubuntu | ✓ |
| CM12 | Linux Mint, Debian | ✓ |
| CM13 | Mozilla | ✓ |
| CM14 | Mozilla | ✓ |
| CM15 | OpenChain | ✓ |
| CM16 | OpenStack, Debian | ✓ |
| CM17 | OpenStack | ✓ |
| CM18 | OSGeo-Live | ✓ |
| CM19 | Perl | ✓ |
| CM20 | PostgreSQL | ✓ |
| CM21 | Python | ✓ |
| CM22 | Ubuntu | ✓ |
| CM23 | Ubuntu | ✓ |
| CM24 | Women who Code | ✓ |
Table 2: Number of Participants Engaged by Contribution Type Based on [38]
| Name | Description | No. |
|---|---|---|
| Source code | Write code, review code, report bugs | 14 |
| Documentation | Write, report issues | 14 |
| Translation | Translate and review translation | 9 |
| Design | User experience design, visual design, style guide creation | 6 |
| Support | Participate in support fora, create cookbooks | 11 |
| Evangelizing | Blog posts, speaking at unrelated events, marketing | 19 |
| Mentoring | Creation of training materials, mentoring contributors | 15 |
| Community management | Participation in working and local groups, conflict resolution, governance | 24 |
| Events | Organization of events, speaking at events | 18 |
| Economic | Make donations and seek sponsors | 12 |
| those concerns. The purpose of this round was to generate a broad overview of the concerns and problems affecting communities. |
Collating this round involved identifying all the unique concerns by name and description, and creating a list of all the unique practices by name, description, and associated concerns.
In the second round, we sought to refine our understanding of both concerns and practices. For the concerns, this entailed collecting information on the prevalence and ranking of concerns, while for the practices we elicited relationships between practices, specifically the preceding/subsequent and complementary relationships, and possible workflows. The collation for this round focused on more elaborate descriptions of practices, and reported on the ranking of concerns. Workflows were also shown.
The third round involved refining the information we had gathered on practices. Participants were asked to verify if they had used or only proposed a practice, and were asked to specify any relationships, context, or limitations which our earlier analyses had missed. The collation consisted of the most extended description of practices.
In each round, questions were posted and participants were given several weeks to respond. At the end of the period, reminders were sent to participants who had not yet responded, and the response time was extended.
After all responses were received, they were analyzed by the lead author using the QDAcity tool for qualitative data analysis. Contextual codes representing the dimensions of interest (community name, participant’s contribution types, and participant’s country) were applied first. Next, the lead author performed theoretical thematic analysis based on the theme of each round [93]. From Round II, the collation was presented to all authors and participants as a collection of practices, also known as a handbook [94]. The collation was sent to participants after each round as a form of member checking [95]. Additionally, after Round III, participants were supplied with a list of practices attributed to them, giving them the opportunity to challenge our interpretation. Participants were given one week to suggest modifications to the collation, then sent the revised document. In the first two rounds we received minor requests for changes, while in the final round we received only acknowledgements of receipt.
Responses to each round were anonymized and then sent to the respondents to confirm that the modifications did not obscure the message. Analysis was conducted on the original responses, but the anonymized responses were used to provide quotations for the collations. Quotations were attributed to individual study participants by means of an assigned two-letter code. Each participant was able to identify their own contributions, and could also build up an impression of other study participants as individuals, without knowing their identities.
4 Results
This section presents the results of our study. Section 4.1 discusses concerns associated with managing episodic contributors. Section 4.2 focuses on the practices for managing episodic contributors, and Section 4.3 extends relationships between practices into workflows.
4.1 Concerns With Episodic Volunteering
We identified a set of concerns that community managers have about EV. Broadly, community managers have a number of concerns about knowledge transmission between the community and episodic participants, the suitability of episodic contributors for tasks, how effectively community processes support EV, and how episodic contributors are included in the community. We identified sixteen concerns that community managers identified regarding episodic volunteering in their communities. Table 3 specifies all sixteen concerns by category, how frequently they were observed, and how many participants ranked these concerns in their top three most pressing concerns.
Space limitations preclude us from discussing all concerns. We illustrate the most common concerns in more detail below. The complete set of concerns is described in the appendix [84], available in the online supplemental material.
Concern 2.C Episodic contributor lacks awareness of opportunities to contribute was deemed most important, observed by 20 community managers and ranked as the most pressing concern by eight study participants. One community manager expressed this urgency as follows:
“Keeping volunteers interested by openly sharing opportunities where they can contribute (technical or non-technical) should be given priority.” —CM
Concern: 2.C Episodic contributor lacks awareness of opportunities to contribute
Communicating opportunities to get involved in a way that reaches episodic contributors is a concern for communities, especially when the people who are aware of tasks which could be done episodically do not enjoy outreach activities. TABLE 3 Concerns by Category, Number of Community Managers Observing Concern, Number of Times Ranked as Most Important Concern, Second Most Important Concern, and Third Most Important Concern
| Concern | Obs. No. | No. #1 | No. #2 | No. #3 |
|---|---|---|---|---|
| Knowledge exchange | ||||
| 1.C Episodic contributor lacks knowledge of developments during absences | 10 | 1 | 1 | 1 |
| 2.C Episodic contributor lacks awareness of opportunities to contribute | 20 | 8 | 1 | 4 |
| 3.C Community lacks knowledge of availability of episodic contributors | 15 | 2 | 1 | 2 |
| 4.C Episodic contributor lacks understanding of project vision | 11 | 1 | 2 | 1 |
| 5.C Episodic contributor and community have mismatched expectations | 13 | 1 | 1 | 1 |
| Suitability of episodic contributors for the work | ||||
| 6.C Episodic contributor quality of work is insufficient | 9 | 2 | 0 | 0 |
| 7.C Episodic contributor’s timeliness and completion of work is poor | 14 | 1 | 1 | 1 |
| 8.C Community’s cost of supervision exceeds benefit of episodic contribution | 8 | 1 | 1 | 1 |
| Community processes do not support EV | ||||
| 9.C Community cannot retain episodic contributors for sporadic requirements | 8 | 0 | 1 | 2 |
| 10.C Community has difficulty identifying appropriate tasks for episodic contributors | 15 | 1 | 4 | 2 |
| 11.C Community lacks an episodic strategy | 14 | 2 | 6 | 1 |
| 12.C Community insufficiently supports episodic contributors | 4 | 0 | 0 | 0 |
| Marginalization of episodic contributors | ||||
| 13.C Community restricts episodic contributors from leadership roles | 12 | 1 | 1 | 1 |
| 14.C Community excludes episodic contributors from discussions and decisions | 10 | 2 | 0 | 3 |
| 15.C Community gives episodic contributors reduced access to opportunities and rewards | 5 | 0 | 0 | 0 |
| 16.C Community lacks appreciation for and recognition of episodic contributors | 9 | 0 | 1 | 1 |
A key characteristic of episodic volunteers is that they contribute irregularly and the nature of their participation tends to be of short duration. This lack of day-to-day engagement with a project means that episodic volunteers may simply not be aware of the opportunities to contribute.
Fifteen community managers observed 3.C Community lacks knowledge of availability of episodic contributors, and two considered it their primary concern. One community manager described the issue for in-person events such as conferences:
“This [lack of knowledge] is a big problem when working with online communities, but it can grow exponentially when you are working a live event. You may do a call for volunteers, and you may end up short-handed, and doing three things at once.” —CM23
This concern directly links to one of the defining characteristics that sets episodic volunteers apart from habitual volunteers. The scenario outlined in the quote above clearly identifies a key issue with episodic volunteers, namely that their availability tends to be much more restricted. In fact, between episodes of activity, these volunteers may be quite removed from what is happening in a community on a day-to-day basis.
Concern 7.C Episodic contributor’s timeliness and completion of work is poor was mentioned by 14 community managers, with one ranking it as the biggest concern. CM24 summarized the concern:
“The main problem of using this kind of help is that sometimes you don’t know whether a person that has started a task is able to finish it all or finish it with a decent quality.” —CM24
Concern: 7.C Episodic contributor’s timeliness and completion of work is poor
Episodic contributors may have less investment in ensuring that their work is completed in a timely manner, or is completed at all. This can be especially problematic if the work is important and others are relying on it. In a situation such as an event, it may be unavoidable to put responsibility on episodic participants.
This concern alludes to the asymmetry of information possessed by community managers and episodic contributors concerning the contributors’ intentions. While contributors are generally aware of their progress and the extent of their dedication to the task, this information is often not conveyed to community managers. For community managers, it becomes difficult to rely on work being completed, or completed to a sufficient standard. With an episodic contributor the problem can be more pronounced, because the community manager may be unable to form an expectation on the quality of future work based on previous experience with the contributor’s work.
CM6 explained why 10.C Community has difficulty identifying appropriate tasks for episodic contributors is a concern. Fifteen community managers had experience with this issue, and one thought it was the most important concern.
“You need to know the context and background for each task to be effective and not get lost. The problem is that to prepare this information usually requires more time than doing the task itself, so normally the person with the knowledge is the one that will do it. It ends up with few people doing a lot of work and possible contributors without knowledge of how to help.” —CM6
Concern: 10.C Community has difficulty identifying appropriate tasks for episodic contributors
Community managers find it difficult to identify and maintain a list of suitable tasks. It can be time-consuming to describe tasks so that they can be picked up by episodic contributors.
It is recommended that episodic contributors be given stand-alone tasks, which can be accomplished without a deep
| Conf. Code | Name | Description |
|---|---|---|
| Community Governance | ||
| ✓ G.1 | Manage the delivery triangle | Adjust scope (quality or features) or schedule when project releases cannot be completed on schedule at the desired level of quality with the expected features. |
| ✓ G.2 | Use longer delivery cycles | Make release cycles longer in order to give episodic contributors the opportunity to contribute without intense time pressure. People who have multiple responsibilities will be able to participate in the project. |
| ✓ G.3 | Host in-person meetings | Host in-person meetings for creative or organizational work involving multiple volunteers. The frequency of meetings may vary by project: it could be yearly, quarterly, monthly, or even more frequent. |
| ✓ G.4 | Make decisions in public | Ensure that decisions are made in a process which is both public and open to suggestions from contributors. Even if the decision is ultimately made by an authoritative body, the transparency of the process can make participants feel a part of it. |
| ✓ G.5 | Create a community definition of quality | Create a community definition of quality so that episodic contributors will know what quality is expected. |
| ✓ G.6 | Craft a community vision | Craft an inclusive community vision and a code of conduct. A clear vision statement helps people determine if they want to participate in the community. |
| ✓ G.7 | Define measuring and success | Define what successful engagement of episodic contributors looks like. Describe how you will measure the impact. |
| G.8 | Centralize budgeting of sponsorships | Centralize the processing of sponsorships and reimbursements so that all claims will be processed in the same manner, and processing will be timely. |
| G.9 | Use an external provider for sponsorships | Hire an external service provider to serve as an intermediary in providing sponsorships. |
| G.10 | Make your leadership diverse | Try to have a diverse board or coordination group to review processes and ensure that they are welcoming and accessible. |
| G.11 | Seek sponsorship | Look for a stable sponsor to ensure continuity of events. |
| Community Preparation | ||
| ✓ P.1 | Identify appropriate tasks | Episodic participants can more easily join if tasks are available. Identify the types of tasks which are suited for episodic contributors. |
| ✓ P.2 | Define one-off tasks | Create stand-alone, one-off tasks. |
| ✓ P.3 | Crowdsource identifying appropriate tasks | Engage experienced contributors in a short-term initiative to identify outstanding issues which could be handled by episodic contributors. Encourage them to continue to identify new tasks, once the backlog has been addressed. |
| ✓ P.4 | Document general working practices | Document the community’s working practices, placing particular emphasis on those areas which are most likely to be relevant to new and episodic contributors, and where contributions will be most appreciated. |
| ✓ P.5 | Detail how to complete a task | Do not just summarize tasks, but detail the steps that need to be taken, and consider providing a time estimate for the task. |
| ✓ P.6 | List current areas of activity | Prioritize tasks and tag them as entry level where appropriate. Group similar tasks together. |
| ✓ P.7 | Hold open progress meetings | Hold regular open meetings where previous work is summarized, and new tasks are assigned. |
| ✓ P.8 | Create working groups with a narrow focus | Create specialized working groups that people can identify with. |
| ✓ P.9 | Create written records of activity | Maintain a summary, for instance in the form of a newsletter, which describes the key discussions and resolutions which took place during a given period. Alternately, rely on written communications (mailing lists, chats) or provide meeting minutes. |
| ✓ P.10 | Keep communication channels active | Ensure that communication channels both online and offline are monitored, and that queries are directed to appropriate people. Make sure that people receive responses. |
| ✓ P.11 | Send ambassadors to small events | Send ambassadors to attend smaller events, to enable personal interactions with potential participants. |
| ✓ P.12 | Respond to all submissions | Respond to every submission in a timely manner. |
| ✓ P.13 | Have a social media team | Recruit people who enjoy social media specifically for the task of communicating with potential and episodic contributors. |
| ✓ P.14 | Set expiration dates | Set distinct deadlines for initiatives. |
| ✓ P.15 | Create continual points of entry | Create ongoing ways for people to join the project and contribute, rather than providing only specific times or times in the process when people can join. |
| Conf. Code | Name | Description |
| ----------- | ------ | ------------- |
| P.16 | Share success stories | Share stories about outstanding or long-serving community members and the challenges they faced and benefits they received. |
| P.17 | Provide templates for presentations | Create one or more standard slide decks which your contributors can use with or without modification. |
| P.18 | Write modular software | Ensure that software is modular. |
| P.19 | Educate sponsoring organizations | Educate sponsoring organizations about participation in open source projects, including topics such as the necessity of maintenance and the open model of production. |
| P.20 | Offer a consistent development environment | Document the workflow, architecture of the module, and use a container to build your project in order to allow people to easily build a local system. Decide upon one recommended way to set up a development environment and focus on this in the documentation. |
Onboarding Contributors
| Conf. Code | Name | Description |
|---|---|---|
| O.1 | Learn about the experience, preferences, and time constraints of participants | Ask new and infrequent contributors about their expectations, availability, preferences and experience. |
| O.2 | Screen potential contributors | Screen potential contributors to determine if they are a good match for the role. This may include having availability at the appropriate time, or being able to commit to a certain amount of time. |
| O.3 | Guide people to junior jobs | Guide people to junior jobs when they do not know where to start. |
| O.4 | Give a choice of tasks | Give participants a choice of the task, from a small number offered to them. |
| O.5 | Manage task assignments with an application | Use an application, such as a wiki or bug tracking system, to handle the assignment process. |
| O.6 | Explain the need for maintenance | Educate contributors about what happens to a contribution after it is included in the project. Explain the benefits to the project if they remain available to maintain their contribution. |
| O.7 | Offer guided introductory events | At events, offer walk-through tutorials on getting started as a contributor, culminating in a hackathon working on a specific beginner problem. |
Working with contributors
| Conf. Code | Name | Description |
|---|---|---|
| W.1 | Have a key contributor responsible | For every important project, make sure that one key contributor is responsible for managing it and responding to inquiries. |
| W.2 | Issue reminders | Send a reminder as the deadline approaches. Be persistent in following up on deliverables. |
| W.3 | Give permission to quit a task | Give people permission to skip on period or task, without recrimination. |
| W.4 | Encourage people to quit | Encourage people who no longer wish to fulfill a role or complete tasks to step down. |
| W.5 | Automate checking the quality of work | Utilize advances in continuous integration/continuous delivery to automate routine evaluation. |
| W.6 | Set expectations | Set expectations for deliverables and communication, even if these are minimal. |
| W.7 | Reject contributions of insufficient quality | Decline contributions which are inappropriate or not of sufficient quality. |
| W.8 | Mentor to quality | Provide mentoring when contributions are rejected due to insufficient quality. This might include access to tools to help people meet quality requirements. Ensure that contributors can always reach out to mentors to get up to speed. |
| W.9 | Require documentation as part of the submission | Require people to sufficiently document their submissions before they are accepted. |
| W.10 | Encourage learners to mentor | Engage episodic contributors in leading other episodic contributors. Let them review episodic contributions and mentor episodic contributors. |
| W.11 | Explain the context of the contribution | Understanding the larger context requires time that not all episodic contributors are able or willing to give. |
| W.12 | Sever ties | Publicly sever the group’s connection to the individual and explain the reasoning. |
| W.13 | Automate process assistance | Consider automation to help people work through the early processes, such as a chat bot or step-by-step interactive site. |
Contributor Retention
| Conf. Code | Name | Description |
|---|---|---|
| R.1 | Publicize your release schedule | Publish your development and release schedule and notify contributors of upcoming milestones, to allow them to plan their engagement. |
| R.2 | Encourage social connections | Encourage people to work together in a small group to accomplish a task. This might also include groups within a company, who can use a... |
| TABLE 4 | ||
| (Continued) |
| Conf. Code | Name | Description |
|---|---|---|
| R.3 | Follow up on contributors | Keep in touch with contributors, even if just by sending an email. |
| R.4 | Instill a sense of community | Help people to understand the cooperative values that underlie free and open source software. This is best done by leading through example. |
| ✓ R.5 | Acknowledge all contributions | Have someone responsible for recognizing returning episodic contributors. This person could thank episodic contributors for returning, or alternately, explicitly welcome new contributors. |
| ✓ R.6 | Reward participation | Offer a tangible reward for participation, such as an organizer’s dinner or swag. Alternatively, offer recommendation letters, certificates, or online recommendations. |
| ✓ R.7 | Recognize everyone | Make use of systems such as badges to recognize the variety of different contributions people can make. At the conclusion of a cycle, thank and identify contributors. |
| ✓ R.8 | Praise publicly | Praise volunteers publicly. |
| ✓ R.9 | Provide evaluations and a promotion path | Provide assessment and opportunities to episodic contributors. Examples of assessment are skill exploration and personal evaluation. Examples of opportunities are travel, employment consideration, succession planning, and skill building. |
| R.10 | Promote episodic contributors | Give sustained episodic participants access to rotating leadership positions which depend on experience rather than continuous contributions. |
| ✓ R.11 | Announce milestones and celebrate meeting goals | Announce when milestones have been met, and celebrate success. |
| ✓ R.12 | Listen to suggestions | Allow anyone who participates to propose what want to implement, even if the decisions are ultimately made by a steering committee. If concepts don’t fit in with the primary project goals, allow people to create unofficial initiatives, provided these don’t damage the project. Invite creators of unofficial initiatives to incorporate them in the main project if they are successful and of high quality. Alternatively, if the project is stand-alone, recognize these successes within the project. Rotate between different focus areas with a consistent schedule. |
| ✓ R.13 | Incorporate unofficial successes | Invite creators of unofficial initiatives to incorporate them in the main project if they are successful and of high quality. Alternatively, if the project is stand-alone, recognize these successes within the project. Rotate between different focus areas with a consistent schedule. |
| ✓ R.14 | Rotate focus areas on schedule | Rotate between different focus areas with a consistent schedule. |
It is only in recent years that many FLOSS communities have sought to create strategies for particular aims, such as retaining newcomers or recognizing non-code contributions. Managing episodic contributors also benefits from a recognition of the problem, identification of the desired outcome, and an evaluation of practices which might be used to achieve the goal. In our previous study, community managers didn’t report making use of any practices for managing EV [6]. This study shows that FLOSS communities are adopting or adapting practices for managing EV. The fact that the concern of how to manage EV effectively remains a high concern demonstrates the need for a study such as ours, which collects and codifies the experience of multiple community managers to create a larger body of knowledge.
4.2 Practices for Managing Episodic Volunteering
We organized the identified practices into a number of categories based on the “lifecycle” of episodic contributors’ engagement. In practice, a community will not address these categories sequentially, but will move between them, iterate through them, or use practices in parallel. However, organizing the practices in categories can help to communicate them to FLOSS community managers. Each practice is aimed at ameliorating one or more of the concerns described in the previous section.
In total, we identified 65 practices in our study across the five categories. Table 4 provides a complete list of practices, along with a brief description of each practice. Of the 65 practices, 48 were confirmed (indicated by a checkmark) to be in use by at least three community managers for the specific purpose of managing EV. The remaining 17 practices were proposed by our panel experts for EV management; they were used by zero, one, or two community managers.
Table 4 contains a brief description of each practice. The full description of each practice is more detailed. In the following subsections, we include as exemplars the full descriptions of one confirmed practice from each category, which was not previously described in the literature (see Table 5). The full descriptions of all practices can be found in the appendix [84], available in the online supplemental material.
The full description of a practice includes the context which may limit the generalizability of the practice, a list of the concerns involved, and a solution. It can optionally include challenges which may arise with implementing the solution, a list of community managers participating in the study who have used the practice, and a list of community managers who suggested but have not used the practice. Additionally, each practice can include a list of related practices. For the most part, practices are not meant to be used in isolation, but to be combined with related practices. Section 4.3 provides examples of how practices can be combined. Relationships between practices can take the following forms, all of which are shown in at least one of the exemplar practices chosen to demonstrate them:
- General/Specific describes a relationship where the specific practice is a more restricted and specialized practice, compared to the general practice. It is demonstrated in R.9 Provide evaluations and a promotion path (a general practice) and O.2 Screen potential contributors (a specific practice).
- Alternative describes two or more practices which address the same concerns with largely incompatible solutions. An example of this relationship is shown in P.8 Create working groups with a narrow focus.
- Preceding/Succeeding is a relationship where practices are best applied in sequential order. An example of this relationship is found in G.5 Create a community definition of quality, which shows both preceding and succeeding practices.
- Complementary describes the situation where practices work well when combined with other practices. W.10 Encourage learners to mentor demonstrates this relationship.
4.2.1 Community Governance
The category Community Governance contains practices that address broad questions about how the community operates. These are practices that will affect the potential episodic contributor’s first impressions of what kind of community it is. One example of practices in this category is G.5 Create a community definition of quality. CM24 stated they were able to make more extensive use of episodic contributors once the community began “documenting our standards of quality.” Another community manager, CM16, explained that new contributors and episodic contributors typically are expected to know what the project considers “quality work,” but that “we never really explain it in a way that’s easy to learn, so it ends up being a barrier to entry.”
Practice G.5: Create a community definition of quality
Context: Episodic contributors do not necessarily know what level of quality is expected. The community is large and mature enough that lack of a common perspective causes problems, and contributors cannot be expected to tacitly acquire the knowledge.
Concerns:
- 4.C Episodic contributor lacks understanding of project vision
- 6.C Episodic contributor quality of work is insufficient
- 7.C Episodic contributor’s timeliness and completion of work is poor
- 11.C Community lacks an episodic strategy
Solution: Create a community definition of quality so that episodic contributors will know what quality is expected. It will become significantly easier to follow many of the subsequent practices if quality is defined within the community.
Related practices:
- P.4 Document general working practices is a COMPLEMENTARY practice.
- G.6 Craft a community vision is a possible PRECEDING step.
- P.10 Keep communication channels active is a possible PRECEDING step.
- P.13 Have a social media team is a possible PRECEDING step.
- G.7 Define measuring and success is a possible SUCCEEDING step.
- P.5 Detail how to complete a task is a possible SUCCEEDING step.
- P.6 List current areas of activity is a possible SUCCEEDING step.
- W.5 Automate checking the quality of work is a possible SUCCEEDING step.
- W.6 Set expectations is a possible SUCCEEDING step.
- W.7 Reject contributions of insufficient quality is a possible SUCCEEDING step.
- W.8 Mentor to quality is a possible SUCCEEDING step.
Challenges: It can be difficult to retroactively apply a definition of quality to an existing project, when not all participants are in agreement.
Used by: CM\textsubscript{15}, CM\textsubscript{13}, CM\textsubscript{14}, CM\textsubscript{18}, CM\textsubscript{24}
Proposed by: CM\textsubscript{16}, CM\textsubscript{19}
4.2.2 Community Preparation
The category Community Preparation contains practices associated with preparing the community to engage episodic contributors. Identifying appropriate tasks and lowering barriers to entry are part of this group. CM\textsubscript{4} explained the reasoning behind practice P.8 Create working groups with a narrow focus to prepare the community for accepting episodic contributors:
“By focusing the working group on a topic that people can identify with, we hope that episodic contributors have an easier time identifying what is useful to them and then have a place to contribute.” —CM\textsubscript{4}
4.2.3 Onboarding Contributors
The category Onboarding Contributors contains practices that can be applied when a new episodic contributor joins the community. O.2 Screen potential contributors is part of the collection of practices for incorporating episodic contributors. A community manager explained why screening can be beneficial:
Practice P.8: Create working groups with a narrow focus
Context: The project is too complex for participants to easily comprehend it in its entirety. It is not possible to readily identify stand-alone tasks in the project.
Concerns:
- 2.C Episodic contributor lacks awareness of opportunities to contribute
Solution: Create specialized working groups that people can identify with. With a narrow focus and defined outcomes, episodic contributors will be able to find tasks more readily.
Related practices:
- P.6 List current areas of activity is a possible ALTERNATIVE step.
- P.18 Write modular software is a possible ALTERNATIVE step.
- P.18 Write modular software is a COMPLEMENTARY practice.
- P.18 Write modular software is a possible PRECEDING step.
- O.1 Learn about the experience, preferences, and time constraints of participants is a possible PRECEDING step.
Challenges: Contributions within the working groups will need to be reported back to the larger group.
Used by: CM\textsubscript{2}, CM\textsubscript{3}, CM\textsubscript{4}, CM\textsubscript{5}, CM\textsubscript{6}, CM\textsubscript{16}
“The first criteria of contribution should be the availability/commitment of participants to donate their time (specifically mentioned as a time frame). This will help reviewers and community leaders to estimate the impact of the contributions.” —CM\textsubscript{14}
4.2.4 Working With Contributors
The category Working with contributors contains practices applied during the period that the episodic contributor is working on an assignment. These practices ensure that episodic contributors’ contributions can be used by the community. A study participant expressed an interest in applying the practice W.10 Encourage learners to mentor when working with contributors:
“It should be possible for the people reviewing episodic contributions to be a different group than the most active developers, so reviews of episodic contributions don’t eat away the time available for other larger contributions. I almost think of this like a mentorship, and the pool of reviewers might even be episodic contributors themselves, who have learned enough to spend part of their limited time on the project reviewing episodic contributions by others.” —CM16
Practice O.2: Screen potential contributors
Context: In order for a contributor to properly perform a role, a certain minimum commitment is required. The project has repeated problems with people insufficiently committing to roles.
Concerns:
- 3.C Community lacks knowledge of availability of episodic contributors
- 4.C Episodic contributor lacks understanding of project vision
- 5.C Episodic contributor and community have mismatched expectations
- 10.C Community has difficulty identifying appropriate tasks for episodic contributors
Solution: Screen potential contributors to determine if they are a good match for the role. This may include having availability at the appropriate time, or being able to commit to a certain amount of time. It is less likely that the commitment will not be met.
Related practices:
- O.1 Learn about the experience, preferences, and time constraints of participants is a more GENERAL practice.
Challenges: Some people will be prevented from pursuing the role, but if there are other forms of contribution it does not prevent them from participating altogether. Assessing potential contributors requires effort.
Used by: CM3, CM8, CM10, CM13, CM14
Another community manager explained how the process can also benefit the mentor:
“Encouraging someone to answer questions on IRC, for example, communicates that you think that they grasp the concepts.” —CM2
4.2.5 Contributor Retention
The category Contributor Retention contains practices that encourage contributors to return. CM13 explained why R.9 Provide evaluations and a promotion path is a useful retention practice:
“It is also important to provide episodic volunteers with metric achievement in the community for their time dedicated and tasks completed. They can grow from basic volunteers to representatives, mentors, influential leaders and even employees, motivating results and retention.” —CM13
Another community manager described an additional benefit for the community:
“[Skills exploration and skill building sessions] can prove helpful as the main goal would be to know what skills episodic volunteers have and what skills they can develop to contribute to more projects (long term or short term).” —CM14
Practice W.10: Encourage learners to mentor
Context: Highly active contributors have limited time to mentor episodic contributors.
Concerns:
- 2.C Episodic contributor lacks awareness of opportunities to contribute
- 4.C Episodic contributor lacks understanding of project vision
- 8.C Community’s cost of supervision exceeds benefit of episodic contribution
- 11.C Community lacks an episodic strategy
Solution: Engage episodic contributors in leading other episodic contributors. Let them review episodic contributions and mentor episodic contributors. Episodic contributors are likely to understand the concerns and limitations of other episodic contributors. Using returning episodic contributors to lead episodic contributors lets core contributors focus on other areas, and recognizes the competency of returning episodic contributors.
Related practices:
- P.16 Share success stories is a COMPLEMENTARY practice.
- W.1 Have a key contributor responsible is a COMPLEMENTARY practice.
- W.8 Mentor to quality is a COMPLEMENTARY practice.
- R.2 Encourage social connections is a COMPLEMENTARY practice.
Used by: CM2, CM5, CM12, CM13
Proposed by: CM11, CM16
Practice R.9: Provide evaluations and a promotion path
Context: Episodic contributors are unable to develop as contributors. There is sustained episodic participation, and absences do not affect the completion of duties.
Concerns:
- 15.C Community gives episodic contributors reduced access to opportunities and rewards
Solution: Provide assessment and opportunities to episodic contributors. Examples of assessment are skill exploration and personal evaluation. Examples of opportunities are travel, employment consideration, succession planning, and skill building. Sustained episodic participants are encouraged to continue contributing and are more beneficial to the community.
Related practices:
- R.10 Promote episodic contributors is a more SPECIFIC practice.
Used by: CM13, CM14, CM22
Proposed by: CM1 4.3 Workflows
Many practices are of limited effectiveness if implemented alone. For instance, it would be impossible to implement O.3 Guide people to junior jobs without first implementing P.1 Identify appropriate tasks, but it would also be ineffective to initiate P.1 without planning to advertise it. However, with a wide range of practices, some tuned to specific contexts, there is no single correct way for a community manager to combine practices to achieve a particular goal.
We asked participants how they might combine practices into a workflow in order to address an important concern. The response to this question can be seen as examples of how community managers approached the task. It is illustrative for other practitioners who wish to understand how to leverage the extensive list of practices that resulted from this study. While it is beyond the scope of this article to identify specific workflows of practices that could be applied to any community—largely due to the fact that communities are only beginning to address EV—the links to related practices within each practice description provide guidance on how community managers have envisioned combining practices.
Each workflow consists of a number of practices, to be implemented sequentially or simultaneously, which together form one possible solution to a specific concern. All workflow diagrams are provided in the appendix [84], available in the online supplemental material.
Fig. 2 depicts an example workflow proposed by CM6 to address concern 11.C Community lacks an episodic strategy. The diagram shows the practices P.1 Identify appropriate tasks and W.1 Have a key contributor responsible as COMPLEMENTARY practices because they are not directly connected to each other, but both PRECEDE practice P.10 Keep communication channels active. P.13 Have a social media team also SUCCEEDS P.1 and W.1.
Another workflow is shown in Fig. 3. It was devised by CM19, and depicts an alternative approach to addressing the same concern. This shows the very individual way in which community managers might join practices to address a concern, based on their own experience and idiosyncratic understanding of their communities.
5 DISCUSSION AND CONCLUSION
5.1 Discussion
5.1.1 Diversity of Practices
In this study we sought to identify the concerns community managers have about episodic volunteers, and identify the practices that they are using—or envisage using—to address these concerns. To do this we conducted a policy Delphi study of community managers.
We looked for study participants engaged in different communities, from different countries, and representing communities of different sizes. In order to identify any relationship between responses based on these dimensions, responses were coded with the community name, countries involved, and activities the community manager had experience with. Observed variations in practices based upon any of the dimensions identified are described in the Context field of the full description of practices.
Community size was an important factor in how episodic contributors are informed about developments. Smaller communities favored a less formal approach such as P.7 Hold open progress meetings while larger communities recommended O.5 Manage task assignments with an application. Mature communities were more concerned with governance and automation practices such as G.5 Create a community definition of quality, W.5 Automate checking the quality of work, O.5 Manage task assignments with an application, and W.13 Automate process assistance.
Country was only associated with one difference. Specifically, reimbursement solutions such as G.8 Centralize budgeting of sponsorships and G.9 Use an external provider for sponsorships were more frequently mentioned in less developed countries, regardless of location. However, it is important to note that the context for these practices is participants who need... sponsorship, and this situation can arise in any country. FLOSS communities had rather consistent concerns and practices around the world and we were unable to observe any cultural differences. Future work might revisit the earlier studies which suggested culture is a factor in FLOSS participation, to determine if this still holds true.
Contribution type produced the greatest amount of diversity in practices. In particular, event organization supplied a number of practices primarily applicable to this context. Software development was another area that stood out as influencing practices. For example, G.3 Host in-person meetings is primarily an event-planning practice, while P.18 Write modular software is clearly specific to software development. Practices specific to one type of work within the FLOSS community were of course less likely to be confirmed than general practices applicable to multiple types of contributions. This may be the reason that some practices, such as P.20 Offer a consistent development environment and P.17 Provide templates for presentations, were not confirmed. Future research could focus on confirming practices for specific aspects of FLOSS work, and on determining the prevalence of their use.
Gender was not directly included in our study design, although participants could introduce gender as context to a problem or solution if they considered it relevant. One participant did mention gender, but as a general statement, noting that women are more responsive to recruitment:
“...in my experience women are more active in volunteering if they find the community responsive. I clearly see the difference in managing gender-related communities and regular communities, that more clearly represent the state of the industry.” —CM24
FLOSS literature suggests that responsive communities are more welcoming to all participants [73], [96], which aligns with the participant’s subsequent statement:
“Making the community friendly for women means making it friendly for everyone who is a kind person, because everyone would feel included and involved. [It’s easy to see if this is succeeding, because women are] literally half of the population.” —CM24
Other ways of increasing female participation include appreciation for diverse teams, tracking of female participation, and improved mentoring [59], [67].
Workflows show another aspect of variation, less easy to quantify. The work of a community manager is “people-centric and versatile,” [97] and it is their implicit and tacit knowledge of their communities which undoubtedly plays a role in determining the construction of a workflow. Future research could try to elicit the factors which go into such decisions.
5.1.2 Comparison to Previous Studies
We identified 65 practices, but we note that this list of practices may not be exhaustive. We compared our findings to an earlier study of onboarding guidelines, which were based on interviews with community managers, diaries of newcomers, and literature [41]. Although their study focused on newcomers, we expected to find overlap because episodic contributors can often only be identified in retrospect [72], not when they join. We also compared our results with our earlier study, where potential practices for managing EV were proposed based on interviews with community managers and the EV literature [6]. Table 5 includes the complete list of practices proposed by the two previous studies, in addition to an overlapping subset of practices from this study.
In total, nine practices appeared in the other studies which were not found in our study. Two practices were identified from the onboarding study [41], and eight from the earlier EV study [6] (one practice was found in both other studies but not our study). Some of this difference can be explained by variable levels of granularity. For instance, Consider time-based releases could be seen as a specific implementation of R.1 Publicize your release schedule. The different research approaches also explain some of the difference. While the previous EV study provided suggestions based on the EV literature, some of these recommendations, such as Evaluate assets, availability and assignments may not be widely-known or systematically applied in FLOSS communities. Still other practices may have been considered so mainstream that participants did not need to mention them, such as Good documentation. In the end, our study identified 52 practices which were not described in the previous studies, in addition to 13 which were previously described (see Table 5). Our emphasis on identifying practices explains why so many new practices relevant to EV were found. Many of these practices are familiar in the FLOSS domain because community managers are adapting existing practices to the EV context.
5.2 Limitations of the Study
The Delphi method is a qualitative method, and so the traditional criteria used for quantitative studies (such as internal validity, external validity, and reliability) are not appropriate due to epistemological differences. Instead, qualitative research is best evaluated by an alternative set of criteria for naturalistic inquiries proposed by Guba [95]. Guba’s criteria are credibility, transferability, dependability, and confirmability.
Credibility. Credibility concerns how plausible, or true, the findings are. Our confidence in the result is strengthened by the fact that the practices were identified iteratively, over a ten month period. This meant that there were many opportunities for participants to reflect on the information which was presented and to amend it. By design, a Delphi study involves member checking during the theory development phase. Preliminary results were also shared with a community manager not involved in the study as an additional form of member checking.
Transferability. Guba recommends purposive sampling as a means of ensuring the transferability of the results [95]. We identified three dimensions which the literature suggested might affect our results and created a diverse Delphi study panel. We were able to observe situations where the dimensions limited the applicability of practices, but were also able to identify broadly applicable practices. We were able to differentiate between novel suggestions and practices which are already in use.
Dependability. Dependability is strengthened by maintaining an audit trail. We maintained anonymized as well as original copies of all responses, including feedback on the collation. We retained a copy of the collation in the state it appeared after each round as well as after feedback was received on the collation. Any supplemental documents developed in creating the collation were also retained in a project repository.
Confirmability. There were multiple opportunities for study participants to correct researcher bias. The multiple phases of a Delphi study allow participants to respond to the developing theory; this is a form of member checking. In addition, we reflected our understanding to participants with a personalized report of practices we understood them to have tried or advocated and requested corrections.
5.3 Conclusion
The identification of 65 practices, 52 of which had not been previously described in the context of managing EV in FLOSS, demonstrates that many community managers are actively thinking about how to incorporate EV. Our study confirms that 74 percent of practices we identified are being actively used. This is in contrast to our earlier qualitative survey on the state of EV in FLOSS communities, where we found that community managers were aware of EV but were not taking any specific steps to manage it [6]. Given the nascent state of the literature on EV in FLOSS communities, this study fills a significant gap. We also described the relationships between practices and gave some examples of how practices can be combined to form a workflow. The findings of this study can be readily adopted by FLOSS community managers.
We further identified 16 concerns that community managers have about EV in their communities, and identified how frequently they were observed by our participants. These concerns were ranked by the expert panel members of this study. The ranked list provides a roadmap for future research as it provides clues as to where researchers and practitioners might direct their energy. Concerns are linked with practices for addressing them, opening the possibility of future studies investigating the effectiveness of different approaches.
With the collection of practices [84] we have created an extensive guide for managing EV in FLOSS which can be readily understood by researchers and practitioners, which draws upon the experiences of seasoned community managers from a number of different communities, geographic regions, and areas of expertise. To the best of our knowledge, this study is the first that has gathered practices for managing episodic contributors in FLOSS communities. Given the increasing attention for episodic contributors as a phenomenon within the open source literature, we believe this study provides a timely foundation for future work in this area.
ACKNOWLEDGMENTS
The authors would like to thank the community mentors who contributed significant time to participate in this study: R. Bowen, N. Bowers, A.-I. Chiuta, S. M. Coughlan, A. El Achêche, B. “bex” Exelbierd, L. Kisuuki, N. Kolokotronis, G. Lelarge, G. Link, S. Park, Pkpacheco, A. Pinheiro, A. Randal, J. A. Rey, C. Shorter, H. Tabunshchyk, L. Vancsa, H. Woo, S. Zacchiroli, V. Zimmerman, and the participants who preferred to remain anonymous. Additionally, we would like to thank the reviewers for their constructive feedback. Finally, S. B. Segletes provided helpful formatting advice. This work was supported, in part, by Science Foundation Ireland grants 13/RC/2094 and 15/SIRG/3293.
REFERENCES
[1] K. Nakakoji, Y. Yamamoto, Y. Nishinaka, K. Kishida, and Y. Ye, “Evolution patterns of open-source software systems and communities,” in Proc. Int. Workshop Princ. Softw. Evol., 2002, pp. 76–85.
[2] A. Mockus, R. T. Fielding, and J. D. Herbsleb, “Two case studies of open source software development: Apache and Mozilla,” ACM Trans. Softw. Eng. Methodology, vol. 11, no. 3, pp. 309–346, 2002.
[3] K. Crowston, H. Annabi, J. Howison, and C. Masango, “Effective work practices for software engineering: Free/libre open source software development,” in Proc. Workshop Interdisciplinary Softw. Eng. Res., 2004, pp. 18–26.
[4] G. Pinto, I. Steinmacher, and M. A. Gerosa, “More common than you think: An in-depth study of casual contributors,” in Proc. 23rd Int. Conf. Softw. Anal. Evol. Reengineering, 2016, vol. 1, pp. 112–123.
[5] A. Lee and J. C. Carver, “Are one-time contributors different? A comparison to core and periphery developers in FLOSS repositories,” in Proc. Int. Symp. Empir. Softw. Eng. Mes., 2017, pp. 1–10.
[6] A. Barcomb, A. Kaufmann, D. Riehle, K.-J. Stol, and B. Fitzgerald, “Uncovering the periphery: A qualitative survey of episodic volunteering in free/libre and open source software communities,” IEEE Trans. Softw. Eng., 2018. [Online]. Available: http://dx.doi.org/10.1109/TSE.2018.2872713
[7] A. Barcomb, K.-J. Stol, D. Riehle, and B. Fitzgerald, “Why do episodic volunteers stay in FLOSS communities?” in Proc. Int. Conf. Softw. Eng., 2019, pp. 948–959. [Online]. Available: https://cora.uc.ie/handle/10468/7248
[8] N. Macduff, “Societal changes and the rise of the episodic volunteer,” Emerg. Areas Volunteering, vol. 1, no. 2, pp. 49–61, 2005.
[9] F. Tang, N. Morrow-Howell, and E. Choi, “Why do older adult volunteers stop volunteering?” Ageing Soc., vol. 30, no. 5, pp. 859–878, 2010.
[10] D. A. Harrison, “Volunteer motivation and attendance decisions: Competitive theory testing in multiple samples from a homeless shelter,” J. Appl. Psychol., vol. 80, no. 3, pp. 371–385, 1995.
[11] R. A. Cnaan and F. Handy, “Towards understanding episodic volunteering,” Vrijwillige Inzet Onderzocht, vol. 2, no. 1, pp. 29–35, 2005.
[12] L. Bao, X. Xia, D. Lo, and G. C. Murphy, “A large scale study of long-time contributor prediction for GitHub projects,” IEEE Trans. Softw. Eng., to be published, doi: 10.1109/TSE.2019.2918536.
[13] J. Gamalielsson and B. Lundell, “Sustainability of open source software communities beyond a fork: How and why has the Libreoffice project evolved?” J. Syst. Softw., vol. 89, pp. 128–145, 2014.
[14] M. Foucault, M. Palyart, X. Blanc, G. C. Murphy, and J.-R. Falleri, “Impact of developer turnover on quality in open-source software,” in Proc. 10th Joint Meeting Found. Softw. Eng., 2015, pp. 829–841.
[15] D. Izquierdo-Cortazar, G. Robles, F. Ortega, and J. M. González-Barahona, “Using software archaeology to measure knowledge loss in software projects due to developer turnover,” in Proc. 42nd Hawaii Int. Conf. Syst. Sci., 2009, pp. 1–10.
[16] M. Zhou and A. Mockus, “Who will stay in the FLOSS community? Modeling participant’s initial behavior,” IEEE Trans. Softw. Eng., vol. 41, no. 1, pp. 82–99, Jan. 2015.
[17] M. A. Hager, “Toward emergent strategy in volunteer administration,” Int. J. Volunt. Adm., vol. 29, no. 3, pp. 13–22, 2013.
[18] N. Macduff, “Episodic volunteers: Reality for the future,” Voluntary Action Leadership, vol. Spring, pp. 15–17, 1990.
[19] K. Culp III and M. Nolan, “Trends impacting volunteer administrators in the next ten years,” J. Volunt. Adm., vol. 19, no. 1, pp. 10–19, 2000.
[20] L. Hustinx and F. Lammertyn, “Collective and reflexive styles of volunteering: A sociological modernization perspective,” Voluntas: Int. J. Voluntary Nonprofit Organizations, vol. 14, no. 2, pp. 167–187, 2003. [21] K. A. Smith, K. Holmes, D. Haski-Leventhal, R. A. Cnaan, F. Handy, and J. L. Brudney, “Motivations and benefits of student volunteering: Comparing regular, occasional, and non-volunteers in five countries,” Can. J. Nonprofit Soc. Econ. Res., vol. 1, no. 1, 2010, Art. no. 65.
[22] R. A. Cnaan, H. Daniel Heist, and M. H. Storti, “Episodic volunteering at a religious megaevent,” Nonprofit Manage. Leadership, vol. 1, no. 1, pp. 1–14, 2017.
[23] S. Koch and G. Schneider, “Effort, co-operation and co-ordination in an open source software project: GNOME,” Inf. Syst. J., vol. 12, no. 1, pp. 27–42, 2002.
[24] T. T. Dinh-Trong and J. M. Bieman, “The FreeBSD project: A replication case study of open source development,” IEEE Trans. Softw. Eng., vol. 31, no. 6, pp. 481–494, Jun. 2005.
[25] J. J. Davies, H. V. K. S. Nussbaum, and D. M. German, “Perspectives on bugs in the Debian bug tracking system,” in Proc. 7th Work. Conf. Mining Softw. Repositories, 2010, pp. 86–89.
[26] F. Rullani and S. Haefliger, “The periphery on stage: The intra-organizational dynamics in online communities of creation,” Res. Policy, vol. 42, no. 4, pp. 941–953, 2013.
[27] D. Riehle, P. Riemer, C. Kolassa, and M. Schmidt, “Paid vs. volunteer work in open source,” in Proc. 47th Hawaii Int. Conf. Syst. Sci., 2014, pp. 3286–3295.
[28] G. Pinto, L. F. Dias, and I. Steinmacher, “Who gets a patch accepted first? Comparing the contributions of employees and volunteers,” in Proc. 11th IEEE/ACM Int. Workshop Cooperative Hum. Aspects Softw. Eng., 2018, pp. 110–113.
[29] A. Capiluppi, K.-J. Stol, and C. Boldyreff, “Exploring the role of community stakeholders in open source software evolution,” in Proc. IFIP Int. Conf. Open Source Syst., 2012, pp. 178–200.
[30] B. Lundell et al., “Addressing lock-in, interoperability, and long-term maintenance challenges through open source: How can companies strategically use open source?” in Proc. IFIP Int. Conf. Open Source Syst., 2017, pp. 80–88.
[31] L. F. Dias, I. Steinmacher, and G. Pinto, “Who drives company-owned OSS projects: Employees or volunteers?” in Proc. V. Work. Softw. Vis. Evol. Maintenance, 2017, Art. no. 10.
[32] L. Dahlander and M. G. Magnusson, “Relationships between open source software companies and communities: Observations from Nordic firms,” Res. Policy, vol. 34, no. 4, pp. 481–493, 2005.
[33] P. J. Ägerfalk and B. Fitzgerald, “Outsourcing to an unknown workforce: Exploring opensourcing as a global sourcing strategy,” MIS Quart., vol. 32, no. 2, pp. 385–409, 2008.
[34] G. Von Krogh and S. Spaeth, “The open source software phenomenon: Characteristics that promote research,” The J. Strategic Inf. Syst., vol. 16, no. 3, pp. 236–253, 2007.
[35] K. Carillo, S. Huff, and B. Chawner, “What makes a good contributor? Understanding contributor behavior within large free/open source software projects—A socialization perspective,” The J. Strategic Inf. Syst., vol. 26, no. 4, pp. 322–359, 2017.
[36] C. Jensen and C. Boldyreff, “Role migration and advancement processes in OSSD projects: A comparative case study,” in Proc. 29th Int. Conf. Softw. Eng., 2007, pp. 364–374.
[37] Y. Fang and D. Neufeld, “Understanding sustained participation in open source software projects,” J. Manage. Inf. Syst., vol. 25, no. 4, pp. 9–50, 2009.
[38] D. Rozas, “Self-organisation in commons-based peer production, Drupal: ‘The drop is always moving’,” Ph.D. dissertation, University of Surrey, Guildford, U.K., 2017. [Online]. Available: https://davidrozas.cc/phd
[39] M. Osterloh and S. Rota, “Open source software development—just another case of collective invention?” Res. Policy, vol. 36, no. 2, pp. 157–171, 2007.
[40] R. Pham, L. Singer, and K. Schneider, “Building test suites in social coding sites by leveraging drive-by commits,” in Proc. Int. Conf. Softw. Eng., 2013, pp. 1209–1212.
[41] I. Steinmacher, C. Treude, and M. A. Gerosa, “Let me in: Guidelines for the successful onboarding of newcomers to open source projects,” IEEE Softw., vol. 36, no. 4, pp. 41–49, Jul./Aug. 2019.
[42] D. Sholler, I. Steinmacher, D. Ford, M. Averick, M. Hoye, and G. Wilson, “Ten simple rules for helping newcomers become contributors to open source projects,” PLoS Comput. Biol., vol. 15, no. 9, 2019, Art. no. e1007296.
[43] K. Crowston and J. Howison, “The social structure of free and open source software development,” First Monday, vol. 10, no. 2, 2005.
[44] K. R. Lakhani, “The core and the periphery in distributed and self-organizing innovation systems,” Ph.D. dissertation, Massachusetts Institute of Technology, Cambridge, MA, 2006.
[45] R. Krishnamurthy, V. Jacob, S. Radhakrishnan, and K. Dogan, “Peripheral developer participation in open source projects: An empirical analysis,” ACM Trans. Manage. Inf. Syst., vol. 6, no. 4, pp. 14–45, 2016.
[46] P. Setia, B. Rajagopalan, V. Sambamurthy, and R. Calantone, “How peripheral developers contribute to open source software development,” Inf. Syst. Res., vol. 23, no. 1, pp. 144–163, 2012.
[47] J. Wang, “Survival factors for free open source software projects: A multi-stage perspective,” Eur. Manage. J., vol. 30, no. 4, pp. 352–371, 2012.
[48] B. Vasilescu, A. Serebrenik, M. Goeminne, and T. Mens, “On the variation and specialisation of workload - A case study of the Gnome ecosystem community,” Empir. Softw. Eng., vol. 19, no. 4, pp. 585–1008, 2014.
[49] G. Von Krogh, S. Spaeth, and K. R. Lakhani, “Community, joining, and specialization in open source software innovation: A case study,” Res. Policy, vol. 32, no. 7, pp. 1217–1241, 2003.
[50] L. Dahlander and S. O’Mahony, “Progressing to the center: Coordinating project work,” Organization Sci., vol. 22, no. 4, pp. 961–979, 2011.
[51] C. Amrit and J. van Hillegersberg, “Exploring the impact of sociotechnical core-periphery structures in open source software development,” J. Inf. Technol., vol. 25, no. 2, pp. 216–229, 2010.
[52] K. Neuling, A. Hannemann, R. Klamma, and M. Jarke, “A longitudinal study of community-oriented open source software development,” in Proc. Int. Conf. Adv. Inf. Syst. Eng., 2016, pp. 509–523.
[53] A. Capiluppi and M. Michlmayr, “From the cathedral to the bazaar: An empirical study of the lifecycle of volunteer community projects,” in Proc. IFIP Int. Conf. Open Source Syst., 2007, pp. 31–44.
[54] H. Masmoudi, M. den Besten, C. de Loupy, and J.-M. Dalle, “Peeling the onion,” in Proc. IFIP Int. Conf. Open Source Syst., 2009, pp. 284–297.
[55] G. Von Krogh, S. Haefliger, S. Spaeth, and M. W. Wallin, “Carrots and rainbows: Motivation and social practice in open source software development,” MIS Quart., vol. 36, no. 2, pp. 649–676, 2012.
[56] A. Lee, J. C. Carver, and A. Bosu, “Understanding the impressions, motivations, and barriers of one time code contributors to FLOSS projects: a survey,” in Proc. 39th Int. Conf. Softw. Eng., 2017, pp. 187–197.
[57] A. Labuschagne and R. Holmes, “Do onboarding programs work?” in Proc. 12th Work. Conf. Mining Softw. Repositories, 2015, pp. 381–385.
[58] I. Steinmacher, M. A. G. Silva, M. A. Gerosa, and D. F. Redmiles, “A systematic literature review on the barriers faced by newcomers to open source software projects,” Inf. Softw. Technol., vol. 59, pp. 67–85, 2015.
[59] S. Balalí, I. Steinmacher, U. Annamalai, A. Sarma, and M. A. Gerosa, “Newcomers’ barriers... is that all? An analysis of mentors’ and newcomers’ barriers in OSS projects,” Comput. Supported Cooperative Work, vol. 27, pp. 679–714, 2018.
[60] C. Mendez et al., “Open source barriers to entry, revisited: A sociotechnical perspective,” in Proc. Int. Conf. Softw. Eng., 2018, pp. 1004–1015.
[61] S. Bayati, “Understanding newcomers success in open source community,” in Proc. 40th Int. Conf. Softw. Eng. Companion Proc., 2018, pp. 224–225.
[62] I. Steinmacher, M. A. Gerosa, T. U. Conte, and D. F. Redmiles, “Overcoming social barriers when contributing to open source software projects,” Comput. Supported Cooperative Work, vol. 28, no. 1/2, pp. 247–290, 2019.
[63] I. Steinmacher, G. Pinto, I. Wiese, and M. A. Gerosa, “Almost there: A study on quasi-contributors in open-source software projects,” in Proc. 40th Int. Conf. Softw. Eng. Companion Proc., 2018, pp. 985–1000.
[64] D. Nafus, “‘Patches don’t have gender’: What is not open in open source software projects,” New Media Soc., vol. 14, no. 4, pp. 256–266, 2012.
[65] K. Carillo and J.-G. Bernard, “How many hawks can hide under an umbrella? An examination of how lay conceptions conceal the contexts of free/open source software,” in Proc. Int. Conf. Inf. Syst., 2015. [Online]. Available: https://dblp.org/rec/conf/icis/CarilloB15
[66] D. Nafus, “‘Patches don’t have gender’: What is not open in open source software projects,” New Media Soc., vol. 14, no. 4, pp. 669–683, 2012.
[67] A. Bosu and K. Z. Sultana, “Diversity and inclusion in open source software (OSS) projects: Where do we stand?” in Proc. ACM/IEEE Int. Symp. Empir. Softw. Eng. Mes., 2019, pp. 1–11.
[68] D. Izquierdo, N. Huesman, A. Serebrenik, and G. Robles, “OpenStack gender diversity report,” IEEE Softw., vol. 36, no. 1, pp. 28–33, Jan./Feb., 2019. [68] M. Storey, A. Zagalsky, F. F. Filho, L. Singer, and D. M. German, “How social and communication channels shape and challenge a participatory culture in software development,” IEEE Trans. Softw. Eng., vol. 43, no. 2, pp. 185–204, Feb. 2017.
[69] M. Burnett, A. Peters, C. Hill, and N. Elarief, “Finding gender-inclusiveness software issues with GenderMag: A field investigation,” in Proc. CHI Conf. Hum. Factors Comput. Syst., 2016, pp. 2586–2598.
[70] M. K. Hyde, J. Dunn, P. A. Scuffham, and S. K. Chambers, “A systematic review of episodic volunteering in public health and other contexts,” BMC Public Health, vol. 14, no. 1, pp. 992–1008, 2014.
[71] M. K. Hyde, J. Dunn, C. Bax, and S. K. Chambers, “Episodic volunteering and retention: An integrated theoretical approach,” Nurse Educ. Voluntary Sector Quart., vol. 45, no. 1, pp. 45–63, 2016.
[72] L. M. Bryen and K. M. Madden, “Bounce-back of episodic volunteers: What makes episodic volunteers return?” Queensland University of Technology, Brisbane, Australia, Rep. no. CPNS32, 2006.
[73] I. Steinmacher, I. Wiese, A. P. Chaves, and M. A. Gerosa, “Why do newcomers abandon open source software projects?” in Proc. 6th Int. Workshop Cooperative Hum. Aspects Softw. Eng., 2013, pp. 25–32.
[74] R. D. Safrit and M. V. Merrill, “Management implications of contemporary trends in volunteerism in the United States and Canada,” J. Volunt. Adm., vol. 20, no. 2, pp. 12–23, 2002.
[75] M. Nunn, “Building the bridge from episodic volunteerism to social capital,” Fletcher World Aff., vol. 24, pp. 115–127, 2000.
[76] L. C. P. M. Meijs and J. L. Brudney, “Winning volunteer scenarios: The soul of a new machine,” Int. J. Volunt. Adm., vol. 24, no. 6, pp. 789–799, 2007.
[77] M. Turoff, “The design of a policy Delphi,” Technological Forecasting Soc. Change, vol. 2, no. 2, pp. 149–171, 1970.
[78] N. Dalkey and O. Helmer, “An experimental application of the Delphi method to the use of experts,” Manage. Sci., vol. 9, no. 3, pp. 458–467, 1963.
[79] W. T. Weaver, “The Delphi forecasting method,” The Phi Delta Kappan, vol. 52, no. 5, pp. 267–271, 1971.
[80] H. A. Linstone and M. Turoff, Eds., The Delphi Method: Techniques and Applications, vol. 18. Boston, MA, USA: Addison-Wesley Publishing Company, 2002.
[81] L. E. Miller, “Determining what could/should be: The Delphi technique and its application,” 2006.
[82] K. Conboy and B. Fitzgerald, “Method and developer characteristics for effective agile method tailoring: A study of XP expert opinion,” ACM Trans. Softw. Eng. Methodol., vol. 20, no. 1, 2010, Art. no. 2.
[83] M. F. Krafft, K.-J. Stol, and B. Fitzgerald, “How do free/open source developers pick their tools?: A Delphi study of the Debian project,” in Proc. 38th Int. Conf. Softw. Eng. Companion, 2016, pp. 232–241.
[84] A. Barcomb, K.-J. Stol, B. Fitzgerald, and D. Riehle, “Appendix to: Managing episodic contributors in free/ libre/ open source software communities,” IEEE Trans. Softw. Eng., to be published, doi: 10.1109/TSE.2020.2985093.
[85] C. Okoli and S. D. Pawlowski, “The Delphi method as a research tool: An example, design considerations and applications,” Inf. Manage., vol. 42, no. 1, pp. 15–29, 2004.
[86] K. Q. Hill and J. Fowles, “The methodological worth of the Delphi forecasting technique,” Technological Forecasting Soc. Change, vol. 7, no. 2, pp. 179–192, 1975.
[87] R. Loo, “The Delphi method: A powerful tool for strategic management,” Policing: An Int. J. Police Strategies Manage., vol. 25, no. 4, pp. 762–769, 2002.
[88] A. L. Delbecq, A. H. van de Ven, and D. H. Gustafson, Group Techniques for Program Planning: A Guide to Nominal Group and Delphi Processes. Glenview, IL, USA: Scott Foresman and Company, 1975.
[89] A. Carvalho and M. Sampaio, “Volunteer management beyond prescribed best practice: A case study of Portuguese non-profits,” Personnel Rev., vol. 46, no. 2, pp. 410–428, 2017.
[90] Y. Takhteyev and A. Hilts, “Investigating the geography of open source software through GitHub,” University of Toronto, Toronto, Canada, 2010. [Online]. Available: http://www.takhteyev.org/papers/Takhteyev-Hilts-2010.pdf
[91] J. C. Crotts and S. W. Litvin, “Cross-cultural research: Are researchers better served by knowing respondents’ country of birth, residence, or citizenship?” J. Travel Res., vol. 42, no. 2, pp. 186–190, 2003.
[92] Y. Y. Kim, “Intercultural personhood: Globalization and a way of being,” Int. J. Intercultural Relations, vol. 32, no. 4, pp. 359–368, 2008.
[93] V. Braun and V. Clarke, “Using thematic analysis in psychology,” Qualitative Res. Psychol., vol. 3, no. 2, pp. 77–101, 2006.
[94] D. Riehle, N. Harutyunyan, and A. Barcomb, “Pattern discovery and validation using scientific research methods,” Friedrich-Alexander Universität Erlangen-Nürnberg, Erlangen, Germany, Tech. Rep. CS-2020-01, Mar. 2020. [Online]. Available: https://dirkriehle.com/wp-content/uploads/2020/03/cs-fau-tr-2020-01.pdf
[95] E. G. Guba, “Criteria for assessing the trustworthiness of naturalistic inquiries,” Educ. Technol. Res. Develop., vol. 29, no. 2, pp. 75–91, 1981.
[96] V. Singh and W. Brandon, “Open source software community inclusion initiatives to support women participation,” in Proc. IFIP Int. Conf. Open Source Syst., 2019, pp. 68–79.
[97] H. Mäenpää, M. Munezero, F. Fagerholm, and T. Mikkonen, “The many hats and the broken binoculars: State of the practice in developer community management,” in Proc. 13th Int. Symp. Open Collaboration, 2017, Art. no. 1.
Ann Barcomb received the PhD degree from the University of Limerick, Limerick, Ireland. She is a member of the Open Source Research Group, Friedrich-Alexander University Erlangen-Nürnberg, Erlangen, Germany and Lero–the Irish Software Research Centre. Throughout her career, she has been active in free/libre/open source software, in particular the Perl community. For more information, please visit ann@barcomb.org.
Klaas-Jan Stol is a lecturer with the School of Computer Science and Information Technology, University College Cork, Cork, Ireland, an SFI principal investigator and a funded investigator with Lero—the Irish Software Research Centre. His research interests include research methodology, and contemporary software development approaches. For more information, please visit k.stol@ucc.ie.
Brian Fitzgerald is director of Lero—the Irish Software Research Centre. He holds an endowed chair, the Frederick Krehbiel II chair in Innovation in Business and Technology, University of Limerick, Limerick, Ireland. His research interests include open source software, inner source, crowdsourcing, and agile methods. For more information, please visit bf@lero.ie.
Dirk Riehle received the PhD degree in computer science from ETH Zürich, Zürich, Switzerland. He is a professor of computer science at Friedrich-Alexander University, Erlangen, Germany. He once led the Open Source Research Group, SAP Labs, Silicon Valley, and founded the Open Symposium (OpenSym). He was the lead architect of the first UML virtual machine, blogs For more information, please visit at http://dirkriehle.com and can be reached at dirk@riehle.org.
For more information on this or any other computing topic, please visit our Digital Library at www.computer.org/csdl.