collective/cdsc_examples_repository
18
0
Fork 0
Code

added revisions and changelog for emilia's gender and sharing paper

Browse Source
This commit is contained in:
Benjamin Mako Hill 2019-06-11 17:29:22 -07:00
commit d888aaf36d
4 changed files with 2162 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

248
cscw_changelogs/2018-gender_sharing/gender_sharing-CSCW2018-reviews.txt Normal file
View File

@ -0,0 +1,248 @@
AC review
score 3/5
Confidence
Confident
First Round Overall Recommendation
3 - Maybe acceptable (with significant modifications)
Contribution and Criteria for Evaluation
The authors present an empirical study of how gender and project feedback affects
sharing on the Scratch online community. The most important evaluation criteria
for this kind of work are:
1) Plausibility of the study (soundness of the followed methods)
2) Descriptions and analysis of findings
3) The implications to CSCW theory and/or practice that result from the study
First Round Review from AC (if needed)
Overall, this is a good piece of writing about sharing behaviors in online
communities (in this case, Scratch projects). For anyone familiar with this online
environment, this paper is of high interest. However, I strongly suggest authors
to better situate the context of the study scenario (in particular, the
cooperative and collective nature of the Scratch platform) for those who are not
necessarily familiar with the topic.
All reviewers have stressed the importance of the intended contribution. However,
while this submission has spread a bit of divergence among reviewers, both
externals have agreed that the current manuscript might benefit from more work
during the R&R stage. However, authors should note that papers with this score
distribution often end up being rejected if they do not properly address the main
concerns raised by reviewers.
Coordinator's First-Round Report to Authors
The following points should be clarified and/or reworked for being reconsidered
during the second round:
- The analysis of people's decision choices seems superficial, although well
executed in statistical terms. In that respect 2AC suggests to consider further
factors, such as the characteristics of the projects, the social relationships
among users within the Scratch community, and the potential "negative feedback"
that project creators could receive.
- R1 questions the consideration of the dataset with regard to the dependent
variable of "sharing a project". In particular, he/she raises the issue of how
critical would be this variable in the Scratch community as opposed to other
contexts of informal learning (where this work has situated itself in the
literature). This should be clarified and justified. Furthermore, R2 requires that
authors explain how the dataset was obtained for analysis.
- R1 also misses a discussion on whether there is (or not) a gender gap on the
sharing decision. R1 also indicates that both the abstract and introduction need
to be rewritten to better reflect this idea. I strongly suggest authors to improve
the discussion on this topic, as I also missed it.
- R2 raises several methodological issues that need to be clarified. For instance,
projects are not necessarily shared in the order they are created, it is not clear
how the formal model in Section 5 was derived, and counting auto-saves as a
measure of effort seems overly simplistic.
- R1 suggests authors to clarify the limitations of their study and address some
of them using complementary research methods. Furthermore, R2 questions the age
distribution in the study sample (particularly as reported in table 1),
highlighting that this could account for some of the discrepancies in the
analysis.
By addressing these points, authors would certainly strengthen the value of their
intended contribution and I explicitly hope to learn more about this topic.
Requested Revisions
(blank)
Formatting and Reference Issues
(blank)
----------------------------------------------------------------
2AC review
score 2/5
Confidence
Confident
First Round Overall Recommendation
2 - Probably NOT acceptable
Contribution and Criteria for Evaluation
This paper aims to reveal how children make decisions on sharing the creative
artifacts in online informal learning communities. The contribution is mostly
empirical. My criteria to evaluate the work will focus on the motivation,
empirical study design, and results.
First Round Review
Regarding the sharing behaviors in online communities, there is a bunch of
literature. The paper focus on the community of children has some novelty
regarding the context. The paper is well written. The intended contribution of the
paper is important. However, there are several major weaknesses in the execution
that undermine the paper
The paper does not actually deal with people's decision choices. Most of the paper
is linking sharing behavior with a few demographics factors though the statistical
analyses are fairly well done and sophisticated. So it does not help too much for
us to understand people's behavioral choice.
To fix this problem, more other factors may need to be considered. Particular some
factors that related to the characteristics of the creative artifacts. Since there
is a community, the social relationships among users also help to shape people's
sharing decision.
The paper also does not operationalize the factor "negative feedback." There are
"Loves" for a project as positive feedback. But with the increasingly popular of a
project, the negative feedback in the comments perhaps also increase, which may
lead to the unwillingness of sharing.
To sum up, I do encourage the authors to continue this research, but I don't think
there is enough time in the r&r cycle for them to improve the study.
----------------------------------------------------------------
reviewer 2 review
score 3/5
Confidence
Confident
First Round Overall Recommendation
3 - Maybe acceptable (with significant modifications)
Contribution and Criteria for Evaluation
By using a quantitative approach, this paper attempts to provide empirical
evidence of (1) a gender gap in the decision to share Scratch (creative computing)
projects, and (2) how this gender gap varies across different levels of the
creators' experience and the level of positive feedback received in the past. The
paper also aims to make a methodological contribution by using a novel method to
analyze a longitudinal process of user engagement in a specific action.
First Round Review
The intended contribution is important as it explores gender differences using a
more nuanced approach than prior literature in the field. The submission offers a
compelling argument to understand why a gender gap might appear in the context of
informal learning and, therefore, why it is important to investigate it in an
online setting. The chosen dataset and methods enable a better understanding of
how other factors, such as experience and positive feedback, relate to the size of
a gender gap regarding the decision to publicly share creative projects.
While this submission does quite well at achieving the intended contribution, I
have some concerns and suggestions about it :
1) The hypothesis development is supported by literature in informal learning. The
use of the action of "sharing a project" as the dependent variable seemed adequate
given that it is an important step according to the literature in informal
learning; however, once the dataset is considered, the selection of this variable
turns more questionable. Less than a third of the projects are shared, the data
analysis only considers projects of creators who have shared two or more projects
(thus reducing the dataset size), and the number of "love-its" (positive
feedback) is rather low (range from 0-10). Therefore, I wonder how critical is the
action of "sharing projects" to informal learning in Scratch. Is it possible that
it is less critical than in other contexts of informal learning? Could that also
explain the unexpected results? Could another variable be used as an alternative
dependent variable?
2) Is there a gender gap when considering the decision to share the first project?
This seems to be an essential aspect of understanding the relationship between
gender and sharing projects; however, it seems that the submission does not
present this aspect.
3) Given that the goal of the paper is to better understand the dynamics of the
relationship among gender, feedback, and sharing, the paper would be much stronger
if some of the method's limitations were addressed by using complementary research
methods. This seems particularly necessary given the unexpected results. For
example, is there any other kind of evidence that can give some support to the
proposed explanation of "second album syndrome"? It would also be beneficial to
know whether there are differences across projects' genres and complexity. If it
is known that there are gender differences across those variables, then it seems
necessary to consider such variables in this analysis as well.
4) I think that the paper is generally well written, except for the abstract and
introduction, which do not explain well why it is reasonable to investigate the
gender gap in this context. There is also a complete paragraph that is repeated in
these two sections.
Overall, I think that this is an interesting contribution. I hope the authors can
address my concerns in the R&R phase.
----------------------------------------------------------------
reviewer 3 review
score 3/5
Confidence
Somewhat confident
First Round Overall Recommendation
3 - Maybe acceptable (with significant modifications)
Contribution and Criteria for Evaluation
The authors analyze how gender and project feedback affects project sharing on the
Scratch platform. In order to do this, the authors analyzed data from shared and
unshared Scratch projects created by 1.1 million Scratch users. The data was
analyzed in a stratified manner, separating the data into groups according to the
order in which the projects were shared by their users (all the projects that were
shared first were analyzed together, all the projects that were shared second were
analyzed together, etc.). The authors define three hypotheses related to gender,
experience level and feedback, and their relation to sharing on the Scratch
platform.
If accepted, can the authors include an explanation of how they obtained the
dataset?
First Round Review
The work is interested and well motivated, but I have several issues with the
methodology followed by the authors:
1. The paper talks about "boys and girls". However, table 1 shows that the age
range is [4, 90]. The mean and median fall in the "boys and girls" age range, but
we don't know much about the distribution of the user ages. Have the authors taken
into consideration in their analysis that some of the projects may have been
created by teachers? This may also explain the sharing behavior exhibited by more
experienced users.
2. Projects aren't necessarily shared in the order they are created. It is not
clear if/how this affects the model proposed by the authors (beta_4).
3. It is not clear from the paper how the formal model in section 5 was derived.
4. Using the number of auto-saves as a measure of effort involved in a project
seems overly simplistic.
5. The order in which the authors presented information in section 5 can be
improved.

BIN
cscw_changelogs/2018-gender_sharing/gender_sharing-CSCW2018-revision_summary.pdf Normal file
View File

Binary file not shown.

438
cscw_changelogs/2018-gender_sharing/gender_sharing-CSCW2018-revision_summary.tex Normal file
View File

@ -0,0 +1,438 @@
\documentclass[12pt,letterpaper]{article}
\usepackage[T1]{ fontenc}
\usepackage[utf8x]{ inputenc}
\usepackage{graphicx}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage[breaklinks]{hyperref}
\hypersetup{colorlinks=true, linkcolor=Black, citecolor=Black, filecolor=Blue,
urlcolor=Blue, unicode=true}
\usepackage[english]{babel}
\usepackage[font=footnotesize,labelfont=bf]{caption}
\usepackage[margin=0.8in]{geometry}
\usepackage{parskip}
\usepackage[round]{natbib}
\usepackage{url}
\def\citepos#1{\citeauthor{#1}'s (\citeyear{#1})}
\def\citespos#1{\citeauthor{#1}' (\citeyear{#1})}
\def\todo{{\normalsize\color{BrickRed}{TODO }}}
\def\done{{\normalsize\color{SkyBlue}{DONE }}}
\begin{document}
\title{Revision Summary for ``Gender, Feedback, and Learners' Decisions to Share Their Creative Computing Projects''}
\author{}
\date{}
\maketitle
\vspace{-1.5cm}
First and foremost, we thank the AC and the three reviewers for the time they have spent on our paper and for their detailed and very helpful feedback. As we hope you will see, we have taken this feedback seriously and spent an enormous amount of time and effort making an extensive revision of our manuscript to address the concerns that were raised. In particular, this work has involved new data collection, new analyses, and new results.
We believe that the changes we have made address all of the substantive concerns raised by the reviewers. In the rare situations where we were not able to follow a reviewer's suggestion, we have also documented that here. In particular, we have made changes to address each of the points highlighted by the 1AC in their revision summary.
As part of these revisions, we have expanded our supplement significantly with additional figures and details of our analyses. If accepted to CSCW, this supplement will be uploaded to the ACM DL alongside our paper. We will also upload all the material in the supplement ZIP file to the Harvard Dataverse and include a link to the permanent DOI for this archive in the camera ready version of our paper.
% DONE - added a sentence to this effect in the paragraph on line 34. maybe add something about the length (it's longer now)
% TODO add something about how the new analysis is OK because we've been sitting with this for a while, it's not that different
This document attempts to provide a comprehensive description of our revisions organized into sections on new analyses (§\ref{sec:analyses}), improvements we've made to our manuscript's framing and background (§\ref{sec:framing}), clarifications about our data and methods (§\ref{sec:methods}), and changes we have made that were not requested (§\ref{sec:other}).
While our revised document is now longer than our initial submission, we believe that the revisions described here have greatly improved the manuscript, and we appreciate the time and effort that our anonymous referees have spent on our work.
\section{New analyses}
\label{sec:analyses}
\subsection{Examining other factors that influence sharing}
% 2AC The paper does not actually deal with people's decision choices. Most of the paper is linking sharing behavior with a few demographics factors though the statistical analyses are fairly well done and sophisticated. So it does not help too much for us to understand people's behavioral choice.
% To fix this problem, more other factors may need to be considered. Particular some factors that related to the characteristics of the creative artifacts. Since there is a community, the social relationships among users also help to shape people's sharing decision.
The 2AC urged us to conduct new analysis with additional control variables:
\begin{quote}
More other factors may need to be considered. Particular some factors that related to the characteristics of the creative artifacts. Since there is a community, the social relationships among users also help to shape people's sharing decision.
\end{quote}
% In particular, the 2AC suggested that we take into account characteristics of (a) creative artifacts and (b) social relations between users to more fully model users decisions to share.
These concerns were echoed by the 1AC.
% DONE - MAYBE. add paragraph (here and in the text) that what we really care about is gender and sharing. we understand that things like social behavior and types of artifacts are going to be correlated with gender and sharing....
% UNSURE I added a sentence about this in the revised abstract, but not sure if more needs to be done than this. [emilia]
R2 also made a point along these lines and suggested that we might consider project type and complexity:
\begin{quote}
It would also be beneficial to know whether there are differences across projects' genres and complexity. If it is known that there are gender differences across those variables, then it seems necessary to consider such variables in this analysis as well.
\end{quote}
To address these concerns, we have collected new data in the form of three new control variables:
\begin{itemize}
\item \emph{Follow Count}: the number of users followed by the user in question at the point that their project was created
\item \emph{Block Count}: a count of the number of programming blocks used in each project
\item \emph{Unique Number of Media Assets}: the sum of the counts of the number of unique audio files and image files used within each project
\end{itemize}
We believe that these additional controls address a closely related concern raised by R3 that, ``using the number of auto-saves as a measure of effort involved in a project seems overly simplistic.'' Although we cannot measure effort directly, controls for measures of the amount of code, amount of images, and amount of time spent on projects seem like they improve on the treatment of this concept in our previous manuscript enormously.
We have made several changes to introduce these new variables:
\begin{itemize}
\item The new controls have been added to our descriptive statistics table.
\item We have added a full paragraph to our data and measures section to introduce these new control variables and describe them in detail.
\item Because several reviewers' comments asked questions about the distributions of our variables, we have included detailed plots in our supplement which show the distributions and medians for our new variables (as well as our other project-level variables) broken down by stratum and gender.
\end{itemize}
Using these variables, we have estimated a second fitted model (M2 in the paper) that includes these three additional controls.
% DONE go through and edit text to summarize and describe the new model (also add paragraph here)
% DONE go through and add text to describe why we still think the old model is useful and illustrative
% DONE update the table with regression summaries so that it's the new model [sayamindu]
% DONE check the text to make sure that the numbers/interpretation being pulled in are accurate [sayamindu]
The results from these models (as can be seen in Figure 4 in the paper) are largely similar. We noted two substantive differences:
\begin{enumerate}
\item In M1 we estimated that the marginal effect of $\mathit{Is~Female?}$ on $\mathit{Is~Shared?}$ was negative for users' first several projects before growing steadily. Our estimates in M2 follow a similar pattern but are ``shifted up'' so that we now estimate a null marginal effect in users' initial projects.
\item In M1, we estimated a positive relationship between $\mathit{Love~Count}$ and $\mathit{Is~Shared?}$ in initial strata, which quickly decreased in size and stabilized as negative. With the addition of controls in M2, this effect is stable and negative throughout.
\end{enumerate}
Although the substantive takeaways from the vast majority of our estimates are very similar, the results are generally more consistent in that effects for less experienced users now look more like those for more experienced users.
We have made many changes in order to present these results:
\begin{itemize}
\item We have added a full paragraph to our Analytic Plan section after our formal model is introduced to describe the two models and to explain both (a) why we present two models and (b) how we interpret the results differently.
\item Our parameter estimates in Figure 4 show both our old model (M1) and the new model (M2) to allow for comparison.
\item All reported parameters and model predicted probabilities in Figure 5 and 6, reported in Table 2 and mentioned in the text, are drawn from this new model.
\item We have made extensive changes to our Abstract, Introduction, Results, and Discussion \& Conclusion sections to interpret the new results and to interpret the unchanged results from M1 and Figure 3 in terms of the new results in M2.
\item Given the much more consistent negative marginal effect of $\mathit{Love~Count}$ on projects' shared status, we have emphasized our discussion of SAS to explain this effect (see §\ref{sec:sas} of this document).
\end{itemize}
To avoid HARKing\footnote{\url{https://www.ncbi.nlm.nih.gov/pubmed/15647155}} and the problems associated with it, we have left our hypotheses unchanged.
Although these results are new relative to the rest our paper, we want to make it clear to the reviewers that we have spent nearly the entire CSCW R\&R period working with these new models and are confident that these results are robust.
% Additional measures we have considered were strongly correlated with ones we have included.
We also gain confidence from the fact that our full model contains every nearly every measure we have seen used in any quantitative study of Scratch.
% . Although there are other ways of measuring social engagement (for example, commenting and love-iting activity) previous work from \citet{hill_cost_2013} has shown that these works are very strongly correlated. We have spent week working with these results
\subsection{Unpack mechanisms behind surprising results for feedback}
\label{sec:sas}
R2 asked for new analysis to help unpack some of our more surprising results in terms of the negative relationship between positive feedback and sharing:
\begin{quote}
Given that the goal of the paper is to better understand the dynamics of the relationship among gender, feedback, and sharing, the paper would be much stronger if some of the method's limitations were addressed by using complementary research methods. This seems particularly necessary given the unexpected results. For example, is there any other kind of evidence that can give some support to the proposed explanation of ``second album syndrome''?
\end{quote}
We agree that our paper would be stronger had it incorporated a qualitative analysis like an interview study or a vignette study to unpack our surprising results. We also think that conducting a study like this well is a research project and paper unto itself.
Although we have deferred working on such a study at present, doing one in the future is something our research group will consider. %We're also very aware that our paper is already longer than most CSCW papers and includes a voluminous online supplement.
%TODO - I don't think the sentence about the length fits here. If that is our argument for not doing a qualitative analysis, it seems weak. I changed it to a comment about itending this as a future project.
In the absence of another study, we have expanded our discussion of SAS significantly so that it now does three things:
\begin{itemize}
\item We have incorporated a small new analysis that we believe that provides some support for SAS. We argue that if SAS is occuring, there were be a positive correlation between positive feedback and effort spent on projects and that the strength of this relationship will increase as users gain experience in the community.
We provide a non-parametric test of this by computing bivariate Spearman's correlations between $\mathit{Autosave~Count}$ and $\mathit{Love~Count}$ across all strata. Although the correlation is relatively weak ($\rho\sim[0.06,0.17]$), it is consistently positive and statistically significant and grows across create counts over that range. We believe this result is consistent with SAS.
\item We have added text explaining that although we think this provides weak evidence in favor of SAS, further work will be needed to unpack and validate this speculation. We have explained that, ``unpacking this effect further will likely require complementary interpretive methods like interviews or a vignette study to understand users' thought processes in this regard.''
\item We have attempted to more clearly articulate reasons drawn from theory and previous research for why SAS might occur. We bolstered our discussion of \citet{ridgeway_framed_2011} on gender and \citet{brennan_audience_2014} on audience in Scratch. We also add discussion of work on imagined audience in social media by \citet{marwick_i_2011} and work on self-censorship in Facebook by \citet{das_self-censorship_2013} which suggests that self-censorship might be more common among boys than among girls.
\end{itemize}
Although, like R2, we would have preferred a more thorough explanation for this surprising finding, we think there is value to testing theories drawn from the literature, reporting what one finds, and doing one's best to suggest possible explanations and lay out avenues for future work that might provide evidence in favor of, or against, these explanations. Some of our favorite papers raise questions as well as answer them. We have worked to do both in our manuscript in regards to this surprising result.
% DONE Referring to previous research on project types/complexity and gender, but also commenting on how this is a separate (though possibly related issue).
% DONE create/include in supplement: distribution of asset count over strata with medians for each gender + overall media [sayamindu]
% DONE create/include in supplement: distribution of block count over strata with medians for each gender + overall media [sayamindu]
% NOTE: Median is a bit of a mess, so had to revert back to geom_smooth() - will explain when we chat
% DONE create/include in supplement: distribution of follower out-degree over strata with medians for each gender + overall media [sayamindu]
% DONE add paragraphs pointing to the online supplement and suggesting that folks look there for more information -- there is already text to this effect at the end of the first paragraph of the RESULTS section -- I expanded that sentence slightly [emilia].
% DONE add section to limitations on this, explain that here: Qualitative work can happen later.
% DONE The only "evidence" for SAS may be Karen's work (especially if we restrict ourselves to Scratch). - already cited at quite some length
% DONE Andy Begel may have done something similar for other contexts - I'll look it up. [sayamindu?] - does not contain what we hoped it would
% DONE mention that we are controlling for complexity - however, showing correlation between complexity autosave count and will be useful.
% DONE make sure frequentist model referred to at the end of the paper reflects new model [sayamindu]
% DONE update summary statistics section [sayamindu]
\subsection{Age as alternative explanation}
R3 raised concerns about the distribution of ages in our sample and the degree to which it might be affecting our results:
\begin{quote}
The paper talks about ``boys and girls''. However, table 1 shows that the age
range is [4, 90]. The mean and median fall in the ``boys and girls'' age range, but
we don't know much about the distribution of the user ages. Have the authors taken
into consideration in their analysis that some of the projects may have been
created by teachers? This may also explain the sharing behavior exhibited by more
experienced users.
\end{quote}
This concern was also echoed by the 1AC.
First, we apologize for taking the distribution of ages in Scratch for granted and for the confusion this caused!
To address this, we have done two things:
\begin{itemize}
\item We have created a new document that shows a series of 26 histograms that reflect the distribution of users' ages (at the time they signed up) across each of the strata in our sample. We have added this document to the supplemental material that will accompany our paper.
\item We have added a paragraph to our Threats to Validity section that discusses this in some depth. In the new paragraph, we do several things:
\begin{itemize}
\item First, we attempt to clearly describe the threat to both our framing in terms of youth and to the fact that our results might be driven by participation by adults.
\item We describe the aggregate distribution of users' ages and point to the public page with this data on the Scratch website (\url{https://scratch.mit.edu/statistics/}).
\item We refer to the new supplementary material described above and summarize it briefly to explain that this distribution is stable across users with different levels of experience.
\item We explain that we include a control for the linear effect of age in all of our models and that its inclusion does not substantively affect our results
\end{itemize}
\end{itemize}
% DONE add link to Scratch statistics-- QUESTION: What do we want the link to Scratch statistics to emphasize? Something about ages of users? This has already been done (See Footnote 5 in the Threats section). I modified the text in THREATS (4th paragraph) to specifically comment on the location of the PEAK of the age distribution, as well as the preponderance of users in the 8-16 age range [emilia].
% DONE create/include in supplement: distribution of follower age at account creation over strata with medians for each gender + overall media [sayamindu]
% DONE add paragraph discussing the histograms showing most projects are created by kids - perhaps comment on how teachers tend to create simple "sample" projects and tend not to become long term participants in the Scratch community (is there a source on teacher participation on online learning sites? Do teachers generally create teacher accounts? Can we filter out teacher accounts from our dataset?)
% DONE if we are controlling for age, these concerns should be moot, no? added comment to this effect in 4th paragraph of THREATS section [emilia].
\subsection{Negative feedback}
The 2AC raised a concern that negative feedback might be driving our results:
\begin{quote}
The paper also does not operationalize the factor ``negative feedback.'' There are
``Loves'' for a project as positive feedback. But with the increasingly popular of a
project, the negative feedback in the comments perhaps also increase, which may
lead to the unwillingness of sharing.
\end{quote}
This is an excellent point raised by the 2AC and echoed by the 1AC. In fact, we had considered this threat but moved forward without attempting to account for it for two reasons drawn from our personal experience with Scratch. First, we believe that negative feedback is very rare. Second, negative feedback is against the Scratch community rules\footnote{\url{https://scratch.mit.edu/community_guidelines}} and is usually flagged and removed---frequently before users see it.
To better respond to this threat to validity, we sought an empirical basis for this intuition with a small informal qualitative study in which one author coded 100 random comments left on Scratch for positivity and negativity in ways that the coder thought might affect users' motivation to share subsequent projects. The coder attempted to be conservative in their coding. The sampling was done in a way so as to only include comments from the time period used for data collection.
The results of this coding were only two potentially negative comments out of the hundred sampled. The full text of these comments are included below:
\begin{enumerate}
\item ``SHADOW THING CRASHED THE GAME''
\item ``Boring!!!!!!!!!!!!!!!!!!!!!!!!!!''
\end{enumerate}
Although we felt that Comment 1 might interpreted as criticism, it might also be interpreted as a good natured bug report or a simple statement of fact. Comment 2 appeared to us to be clearly negative. Investigation revealed that it had had been flagged and censored. Although we cannot know for sure, it is very possible that the author of the project never saw the only negative comment in our 100 comment sample.
Although the potential demotivating effect of these negative comments provides an empirical basis for concern, the very small proportion (1-2\%) gives us confidence that the presence of negative feedback is unlikely to be driving the overall patterns observed in the data.
% DONE add paragraph to the paper -Added this paragraph to the Empirical Setting section [emilia]
% DONE add section to online supplement adapted from this text - I wasn't sure how to add this to the supplement, so I added it to supplement.tex - BUT IT WILL NEED TO BE PROPERLY ADDED TO THE REAL SUPPLEMENTAL MATERIALS [emilia]
We have added a paragraph to the paper that summarizes the threats, our analysis, and these results. We have also added a section to our online supplement that includes an adapted version of the text we have included in this section.
% DONE consider dealing with this:
% Re. negative feedback, we can point to the Community Guidelines asking for "constructive" and "positive" comments. Trolling, etc. are taken down - however, there's not really a citeable object for that, maybe except for a blogpost that Lombana wrote: https://medium.com/berkman-klein-center/moderation-and-sense-of-community-in-a-youth-oriented-online-platform-scratchs-governance-eeac6941e9c9 - Added to the Empirical Setting section [emilia]
\subsection{Analysis of gender gap for users' first projects}
\label{sec:firstprojs}
R2 asks, ``Is there a gender gap when considering the decision to share the first project?''
% DONE report information on first project (sayamindu?)
Indeed, there is! It was visualized in Figure 3 but not mentioned elsewhere. We understand that it must have been confusing that this was largely missing from our manuscript!
We have made two changes to address this confusion:
\begin{itemize}
\item We have added a new sentence to the paragraph interpreting Figure 3 to report the numbers for the differences in the proportion between boys' and girls' first projects.
\item We have reworked our analytic plan so that it now includes this sentence explaining why these data are not included in our models and/or figures. This sentence says: ``Our models are estimated on a dataset that incorporates data beginning from the 2\textsuperscript{nd} project shared by each user because users must have created at least one project in order for there to be any variation in users' number of previously shared project and/or received love-its.''
\end{itemize}
\section{Improvements to framing}
\label{sec:framing}
\subsection{Emphasizing importance of sharing}
R2 suggest that we needed to spend more time justifying why studying sharing is important (especially in the context of online learning):
\begin{quote}
The hypothesis development is supported by literature in informal learning. The use of the action of ``sharing a project'' as the dependent variable seemed adequate given that it is an important step according to the literature in informal learning; however, once the dataset is considered, the selection of this variable turns more questionable. Less than a third of the projects are shared, the data analysis only considers projects of creators who have shared two or more projects (thus reducing the dataset size),\footnote{This issue is discussed in section §\ref{sec:firstprojs}.} and the number of ``love-its'' (positive feedback) is rather low (range from 0-10). Therefore, I wonder how critical is the action of ``sharing projects'' to informal learning in Scratch. Is it possible that it is less critical than in other contexts of informal learning? Could that also explain the unexpected results? Could another variable be used as an alternative dependent variable?
\end{quote}
R2's point raises a number of important issues and we have made a number of changes to the manuscript to address them:
\begin{itemize}
\item We have edited our manuscript to make it clear that we think that the choice of sharing as a dependent variable is important precisely because it is relatively rare. We believe that if sharing happened all the time, it wouldn't be important to try to understand why it does and doesn't occur.
\item We have added new text and citations to our background section describing several high-profile studies by \citet{das_self-censorship_2013} and \citet{marwick_i_2011} that attempted to understand sharing (and not sharing) in Facebook and Twitter, respectively. We believe that this work can make a much better case for sharing as a dependent variable in social computing research.
% >> Maybe just frame it better - say that this problem exists in other creative communities too (e.g. Deviantart), but the difference here is that we can measure the unshared projects. R1 is also wondering about how "important" sharing is. We should say - it's important, especially when we consider theory, and it's a problem that a small proportion is shared, and our work tries to figure out, for the first time, what may help toward more sharing. For how dataset was obtained - we can reach out to 1AC and tell them that that we are concerned that would compromise the anonymity of the process.
% TODO Mention similar communities (e.g. Deviantart, NewGrounds, etc.)
% TODO cite creepy facebook study (Emilia)
% TODO cite marwick and boyd study on self-censorship (Emilia)
% Note, I am working on the 3 items above -- should be added later today (Monday) [emilia]
% \subsubsection{Sharing and learning}
\item We have heavily edited our background section, adding text to more clearly explain how previous work has suggested that sharing can support learning. We break this down into two possible processes:
\begin{enumerate}
\item We explain that sharing can support learning by the sharer by allowing learners accesss to opportunities for receiving critiques on their projects, serving as an entry into engaging community, eliciting further participation, and providing access to rich interactions with other learners. Some of this argument (and several citations) were incorporated in a more slip-shod way into the previous drafts. We have improved this.
We have added text making this process more comprehensive and explaining, much more clearly, that sharing creates opportunities to engage socially with others in a community in ways that support virtual cycles of social participation. We point to a number of studies that have shown that increased participation will, on average, support increased learning \citep{scaffidi_skill_2011, yang_uncovering_2015} as well as more recent work that has begun to unpack this process \citep{dasgupta_how_2018}. We have also referenced \citet{jenkins_confronting_2009} in our discussion of how participation in communities can lead to increased levels of learning.
We have also added text to connect this directly to Scratch by quoting \citet{brennan_new_2012} who argue unshared projects in Scratch ``might highlight areas of conceptual confusion or challenge'' which might be addressed if the projects were published.
\item Sharing can further support learning by creating valuable material through which others can learn by inspecting and building off others' work \citep{dasgupta_remixing_2016}. Through sharing, learners can build on and extend each others' work \citep{hill_cost_2013, resnick_all_2007}, with improvements in the quality of artifacts constructed \citep{dow_parallel_2010, hill_cost_2013}.
\end{enumerate}
% In several places in the text, we also try to articulate that we believe that sharing is an important, but not essential, pathway to learning. After all, plenty of learning happened in Scratch before 2007 the Scratch online community was launched.
\item Finally, we have edited our text in several places to make it clear that we think that sharing is an important pathway to learning in informal online learning environments, not an essential feature of it.
\end{itemize}
% The 2AC is correct that sharing is rare in Scratch, as it is in other places, but the fact that is rare doesn't mean that it's a useful things to study! Indeed, building toward a better understanding of sharing and the dynamics that drive it (as we do here!) seem like a very step toward supporting increased level of sharing. The fact that it is relatively low (and there is so much variation between users) is precisely why we believe this is an interesting study.
% DONE improve this section to explain what this looks like - Added a paragraph to the INTRODUCTION on learning on Scratch, separate but ultimately dependent on sharing [emilia].
% DONE cite wide walls study, scaffidi, etc, to suggest that measures of some kind of learning.
% DONE Here's a quote from Karen and Mitch's paper that points to unshared projects being a potential problem for people doing analysis:
% We learned through interviews and observations that many young people do not post all of their projects to the Scratch online community. In particular, in-progress projects and abandoned projects were often not posted, or were posted to an alternative, test account. These projects could be particularly interesting from a developmental perspective, as they might highlight areas of conceptual confusion or challenge.
% https://web.media.mit.edu/~kbrennan/files/Brennan_Resnick_AERA2012_CT.pdf
%THOUGHT: These in-progress/abandoned projects are perhaps the projects in greatest need of sharing and most likely to benefit from being viewed and critiqued. The current ``work-in-progress'' designation is a nice start. What percentage of projects are shared this way? When was this instituted? How has the percentage of WIP projects changed over time?
% DONE make it clear that there are multiple pathways that learning can happen in Scratch [emilia] -- added to INTRODUCTION (as part of how sharing contributes to learning, discussed learning outside sharing as well)
% TODO add material into the background emphasizing the cooperative and collective nature
\subsection{Improve discussion of gender in abstract and introduction}
R2 (echoed by the 1AC) explained that, ``I think that the paper is generally well written, except for the abstract and introduction, which do not explain well why it is reasonable to investigate the gender gap in this context. There is also a complete paragraph that is repeated in these two sections.''
Having returned to our manuscript, we agree completely that this was a major weakness and oversight. We have made several changes:
\begin{itemize}
\item We have rewritten our abstract almost completely to address these concerns and to foreground the role of gender in our analysis.
\item We've added a full new paragraph to our introduction that builds up the intuition and importance of studying gender.
\item We have separately rewritten both the abstract and the offending paragraph in the introduction so there should no remaining repetition.
\end{itemize}
We apologize for the earlier state of the draft in this regard. We hope that the reviewers agree that the manuscript is improved as a result.
% DONE Rewrite abstract and fix introduction so it stresses this and is not as repetitive as it currently is. we should carefully reread this section and make changes to try to address this - rewrote abstract, but still needs editing and a "how we contributed" ending [emilia].
\subsection{Emphasize the social and cooperative nature of Scratch}
The 1AC suggested that we ``...better situate the context of the study scenario (in particular, the cooperative and collective nature of the Scratch platform) for those who are not necessarily familiar with the topic.''
We have added several sentences to our Empirical Setting section to try to emphasize this. We have mentioned Scratch's collaborative nature in several places, including in our rewritten abstract.
\section{Clarifications in regards to data and methods}
\label{sec:methods}
\subsection{Explain access to dataset}
R3 asks: ``If accepted, can the authors include an explanation of how they obtained the dataset?''
We have added text to our Data and Measures section explaining that access to data was granted through a collaborative research agreement with the Scratch team.
Access came in the form of permission to query a copy of the SQL database that runs the Scratch online community. It was allowed by having our team members added as researchers on an IRB protocol at MIT that covers \textit{post hoc} analysis of this database.
% DONE add sentence or two above to the paper - Added to second paragraph of 'Data and Measures' section [emilia]
\subsection{Improvements to ``Analytic Plan'' section}
R3 explained that, ``The order in which the authors presented information in section 5 can be improved,'' and that, ``it is not clear from the paper how the formal model in section 5 was derived.''
We have changed the order of material in this section, added new information about the addition of controls, and added a full paragraph that we hopes makes the derivation of the formal logistic regression model more clear. We believe that these changes all reflect improvements to the text.
We have also made clear that the formalism on the left side of the equation is a description of the logistic link function.
% TODO change order of material in Analytic Plan section
% DONE add new information about controls
% PARTIALLY DONE - added number of assets control to sentence in Threats on project complexity and genre. Still need to address number followed control
% DONE add information on derivation of formal model [sayamindu]
\subsection{Creation order versus sharing order}
R3 raised a methodological question about sharing order that was echoed by the 1AC saying, ``Projects aren't necessarily shared in the order they are created. It is not clear if/how this affects the model proposed by the authors ($\beta_4$).''
This is an excellent point and actually describes two related threats in this measure that we had considered: (a) projects may be shared in an order that is different than the order they are created, and (b) projects may be shared at any point in time.
In terms of (a), R3 is correct that projects are not necessarily published in the order they are created and this is a potential threat. Unfortunately, for constructing strata, we felt that this was the only reasonable option because strata must simultaneously include both shared and unshared projects and unshared projects cannot be placed in any ordering based on share order.
This is not to suggest that other orderings are not possible. For example, we might order projects based on the final time a user changed them. That said, because users can always go back and change any of their projects, this would require a fixed window (like the one we use for creating our measure of $\mathit{Autosave~Count}$) which would introduce right censoring and would introduce most of the same concerns related to out-of-orderness.
Although we believe our current measure is the best choice given our data as a result, we agree with R3 that this choice might introduce noise or bias (although we cannot think of a specific mechanism through which this bias might occur).
The second issue (b) that projects may be shared at any point also reflects a potential for noise and bias. The threat is that projects created later in time might have ``less chance'' to be shared. We are not particularly worried about this because most projects are shared very quickly and because we have at least 7 months of data for every project in our dataset which appears likely to capture nearly all of the variation in this measure.
We have added a full new paragraph to our Threats to Validity section that details both of these threats, explains that we cannot rule out noise or bias, and explains why we believe they are likely not driving our results by summarizing the logic given in this section.
We thank R3 for pointing out this issue!
% sdg: I don't think it matters that the orders may be different - R2 may be confused because of the control share_count, and may be assuming that we are incorrectly measuring it. However, we should think through and explain this very carefully. For model derivation, we can circle back to the earlier point about incorporating additional controls and why auto-save counts are good.
% DONE discuss why order of sharing is the only approach that makes sense. Will describe how our model was derived in more detail. Will add features that should correlate with autosaves (i.e. "complexity" measures such as sounds/images and scripts/blocks) as described below [sayamindu?]
% - Address by explaining that this is not something we can actually evaluate.
\section{Other Changes}
\label{sec:other}
We have also made a series of changes that were not requested by the reviewers:
\begin{itemize}
\item We have added two sentences to our analytic plan section explaining why we log transform our variables.
\item We realized that projects on Scratch can be ``self-loved'' (i.e. the author of the project can click on love-it on their own project). While this is a rare practice, we updated our programming code to make sure this is accounted for and self-loves are not counted in the analysis in this updated version. This change did not affect our results at all.
\item We have added a paragraph of text to better describe our prototypical probability plots more effectively. In particular:
\begin{itemize}
\item We have clearly described that this table reflects simulated data generated using our model.
\item We have explained how to interpret Bayesian prediction intervals. We were worried because these are substantially different than credible intervals for parameters or confidence intervals in a frequentist framework.
\item We moved some text from our figure caption into the text to explain how control variables were fixed. We have much more clearly and specifically explained how we have fixed variables that do not vary visibly in the graph.
\end{itemize}
We believe that these changes make interpreting our results easier. We believe this is important given the addition of the requested analyses.
\item We have carefully edited our document for clarity, redundancy, and style. We hope that the document is easier to read as a result.
\end{itemize}
% DONE go through and add information helping folks interpret the the prediction intervals
% DONE mention the way in which we are calculating median values of controls separately for girls and boys while doing prediction
\bibliographystyle{chicago}
\bibliography{references}
\end{document}

1476
cscw_changelogs/2018-gender_sharing/references.bib Normal file
View File

File diff suppressed because it is too large Load Diff