collective/cdsc_examples_repository
17
0
Fork 0
Code

added CSCW 2016 paper on remixing and learning

Browse Source
This commit is contained in:
Benjamin Mako Hill 2019-06-11 17:58:39 -07:00
parent 4fc082a8ac
commit a3aa7637e0
4 changed files with 423 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
cscw_changelogs/2016-remixing_learning/README.txt Normal file
View File

@ -0,0 +1,7 @@
Material for:
Dasgupta, Sayamindu, William Hale, Andrés Monroy-Hernández, and Benjamin Mako
Hill. 2016. “Remixing As a Pathway to Computational Thinking.” In Proceedings
of the 19th ACM Conference on Computer-Supported Cooperative Work & Social
Computing (CSCW ’16), 1438–1449. New York, New York: ACM.
https://doi.org/10.1145/2818048.2819984

BIN
cscw_changelogs/2016-remixing_learning/remixing_learning-CSCW2016-randr_letter.pdf Normal file
View File

Binary file not shown.

151
cscw_changelogs/2016-remixing_learning/remixing_learning-CSCW2016-randr_letter.tex Normal file
View File

@ -0,0 +1,151 @@
\documentclass[12pt,letterpaper]{article}
\usepackage[T1]{ fontenc}
\usepackage[utf8x]{ inputenc}
\usepackage{graphicx}
\usepackage[usenames,dvipsnames]{xcolor}
\usepackage[breaklinks]{hyperref}
\hypersetup{colorlinks=true, linkcolor=Black, citecolor=Black, filecolor=Blue,
urlcolor=Blue, unicode=true}
\usepackage[english]{babel}
\usepackage[font=footnotesize,labelfont=bf]{caption}
\usepackage[margin=0.8in]{geometry}
\usepackage{parskip}
\begin{document}
\title{Revision Summary for ``Remixing as a Pathway to Computational Thinking''}
\author{}
\date{}
\maketitle
First, we want to thank all three reviewers and the AC for their thoughtful and constructive feedback. We are glad that the reviewers found our work relevant and interesting. We have read the reviews carefully, taken the feedback seriously, and made quite a number of revisions to our manuscript in response. We have attempted to address every issue raised in the reviews we received. We feel our paper is much stronger as result. In this document, we describe the changes we have made.
\section{Effect Size (AC, R1, R2)}
The most important issue raised by reviewers was around our inadequate discussion of effect size -- especially given the large size of our data set. Comments included:
\begin{quote}
The biggest concern was small effect size, so you might address this issue further in limitations or discussion. (AC)
I'm slightly concerned that the effect size is small. Papers with a large $n$ and small effect size are worrisome -- does this really mean much? (R1)
Unfortunately, while the analysis confirms the authors' initial hypotheses, with statistical significance (as you might expect from a sample of this size), the effect sizes were quite marginal (as the authors acknowledge themselves) \ldots
Still, the analysis does give initial confirmatory evidence that
remixing is correlated with learning, and contributes methods that
could be built in. For instance, they might be applied in more
focused contexts (without coloring contests), or more formalized
educational interventions, where we larger effect sizes might be
observed. (R2)
\end{quote}
We have made several changes to address this concern. First, we have made several small changes to our results section to clarify our reporting, and interpretation, of effect sizes. Second, we have added a full paragraph to the discussion entirely focused on the size of our effects.
In this new paragraph, we do several things. First, we discuss how considering \emph{Downloads} might affect our interpretation of effect size (see below). Second, as suggested by R2, we point out that small effect sizes may, in part, reflect the informal and unfocused nature of Scratch. To sharpen this point and to suggest ways that designers might support ``high value'' remixing, we add references to work by Brennan on formal learning environments in Scratch, to Cheliotis and Yew on how contests can shape and incentivize remixing in informal environments, and to Roque on how Scratch itself has tried to support ``high value'' remixing through focused interventions in the form of ``collab camps.'' We agree with R2 that effects may be much higher in contexts like these when designers leave the quality and quantity of remixing less up to chance than the designers of Scratch have done.
\begin{quote}
Please remove the word "strong" from the first sentence of the discussion section. The effect sizes are small, and that isn't really "strong" support in my view. (R1)
\end{quote}
We have removed the word ``strong'' from the first sentence of the discussion. We also carefully edited the rest of the paper and removed several other similar qualifiers. We overstated our findings and we apologize.
\section{Robustness Checks (R2)}
R2 suggested that we report more detail on our robustness checks, especially our results on copy and pasting:
\begin{quote}
In the limitations section, the authors discuss copy \& pasting as a
shallow demonstration of CT concept in de novo projects. Perhaps
they should include the results of this adjustment in the results
section. I would also be curious to know the precise extent of copy
\& pasting that they detected. (R2)
\end{quote}
As suggested, we have reported the range of copy and pasted projects (i.e., 2-15\%) in the text of the manuscript.
We considered including the full results of robustness checks for both this test and our stricter model of parallelism but concluded that the addition of 1-2 pages of results with identical implications was hard to justify in a paper that was already 12 pages long.
Instead, we have created an appendix to the document that we have included as supplementary material. In it, we include detailed data on the extent of copying and pasting as well as a version of Table 4 that reflects this adjustment. Although it was not requested, we also included results from our robustness check that limits our analysis to in-sprite parallelism.
If our paper is accepted, we plan to publish this as an online supplement in the ACM Digital Archive. If this is not possible, we will publish our results in our library's institutional archive and include a link to this archival copy in the text of the camera ready version of our manuscript.
\section{Expanded Interpretation of Results (R2)}
R2 suggested that we should improve our interpretation of the controls used in our models and pointed to examples of under-explained variables:
\begin{quote}
More explanation about some the results would be helpful, since some
of them ran against my expectations or intuition. For example, why
remixes have such a strong negative association with particular
concepts (much more negative than the remixes w/ concepts was
positive), and why Downloads is predicted to moderate the effects of
remixing. (R2)
\end{quote}
We have made a series of changes to improve our discussion and interpretation of controls. We expanded upon and added a paragraph to the part of our results section that discusses controls. In particular, we interpret the effects of \emph{Comments} (mixed in sign between our two sets of models) and \emph{Remixes} (as suggested by R2). In terms of the latter, we suggest that we would not expect a positive effect because remixing projects that do not involve a concept seems unlikely to increase the likelihood that a particular concept will be used and that we believe that the negative effect associated with these estimates, although not anticipated, may be explained by social forms of remixing like coloring contests that are imperfectly controlled for by \emph{Total Blocks}.
\subsection{Discussion of Downloads (R2)}
Although downloads were only mentioned briefly in R2's review, we concluded that our interpretation of the measures pointed to a major area for improvement in our manuscript and we have made several changes to improve interpretation of our results in this regard.
% In Scratch 1.4 (the version analyzed in this paper) downloading was the only way for users to view the code of projects and is a necessary step toward remixing. Because it happens every time a project is remixed, it will be positively correlated and its inclusion will likely explain some of the variation explained by remixing. We include both Models 2 and 3 because there is conceptual overlap between downloading and remixing among theorists. Manovich, for example, clearly argues that reading something and becoming inspired \emph{is remixing} and this form of learning is part of what Papert would call ``appropriation.'' We have made a number of changes to clarify our interpretation in these senses.
We have completely rewritten and added to the paragraph on downloads in our \emph{Data and Measures} section in order to emphasize that downloading reflects the only way that users can look inside projects and that downloading has strong conceptual overlap with remixing. We point out that Manovich clearly argues that reading something and becoming inspired constitutes a form of remixing and that learning through downloading is part of what Papert calls ``appropriation.''
We have also added several sentences to our \emph{Results} section explaining why we predict a moderating effect and explicitly interpreting the size of the coefficient associating with \emph{Downloads} in M3. We also reflect on the pattern of results for downloads in the end of that section. Finally, we refer to our results for downloading in the new paragraph on effect size we have added to our \emph{Discussion} section.
\subsection{Concept-by-Concept Interpretation of Results for H2 (R2)}
R2 expressed some dissatisfaction with the lack of insight from concept-by-concept comparisons:
\begin{quote}
In addition, I was hopeful for insights from analyzing the data on a
concept-by-concept level, but there was little gained in comparing
and interpreting the different concepts, given the similar but
marginal effect sizes. (R2)
\end{quote}
% https://github.com/makoshark/scratch-remixlearning/blob/c6708a4cb3ad02bb7055f902982a11cde45dbe70/paper/appropriation-cscw2016.Rnw#L437
Like R2, we would have also loved results that showed more variation in effect size between concepts. Of course, the results are what they are and we are playing the hand we were dealt. That said, reflecting on R2's comment led us to conclude that we could do better in this regard. Toward that end, we noted that, in general, lower estimates were associated with CT concepts that had lower uptake in Scratch. We have pointed this out and suggested that while it is difficult to draw clear conclusions from this, this phenomena may point to a general difficulty in understanding or learning these CT concepts, at least within Scratch.
\section{Stylistic Improvements (AC, R2, R3)}
Finally, we made a series of stylistic improvements to our manuscript.
As suggested by the AC, we fixed the references section to use an up-to-date style template so that it matches the new ACM guidelines. We have fixed all of the typos pointed out by R2 and fixed the reference to Figure 4 as suggested by R3. We have carefully proofread the paper to try to eliminate other errors. If accepted, we will have the paper proofread by a professional before we submit the camera-ready copy.
We thank the reviewers and the AC for all the time and energy they have spent on our paper. We believe our manuscript is much better shape as a result.
% The following typographical errors were fixed:
% Item
% Reviewers’ comments
% Responses
% Typo
% Empirical Setting: "(though *\_\_\_* are more granular)" (R2) added “blocks”
% Typo
% Data and Measures: "...we include a control for the number of comments receive by *an* user" (R2)
% changed to “a”
% Typo
% Limitations: "In line with *out* expectations..." (R2)
% changed to “our”
% Typo
% Discussion: "Although our results *provides* evidence..." (R2)
% changed to “provide”
% \begin{quote}
% I looked in vain for any weaknesses that might undermine confidence
% in the paper's claims or results. Finally, I found one!! I think. In
% the Results section, paragraph 4, you said you showed plots for two
% prototypical users in "Table 4," whereas I think you mean here
% "Figure 4." There. Don't say R3 didn't contribute anything of value…
% (R3)
% \end{quote}
\end{document}
% LocalWords: collab novo Manovich

265
cscw_changelogs/2016-remixing_learning/remixing_learning-CSCW2016-reviews_round1.txt Normal file
View File

@ -0,0 +1,265 @@
------------------------ Submission 423, Review 4 ------------------------
Title: Remixing as a Pathway to Computational Thinking
Reviewer: AC
Overall rating: 4 (scale is 1..5; 5 is best)
Reviewer expertise: 4 (scale is 1..4; 4 = "Expert")
Contribution and Criteria for Evaluation
This paper empirically examines "remixing," or reuse and modification of
others' solutions, in the context of learning coding concepts using MIT's
Scratch environment for young people. Several dependent measures showed
improved and deep learning by remixers.
Expertise
4 (Expert)
Formatting and Reference Issues
The CSCW formatting of references has changed. Please take a look at the
new template (links to Word and Latex templates are available at
http://cscw.acm.org/2016/submit/papers.php) and make the necessary
changes to your reference section.
First Round Overall Recommendation
4 (Probably acceptable (with minor modifications))
First Round Review (if needed)
Coordinator's First-Round Report to Authors
The reviewers agree that your study is well planned, the results are
interesting, and the paper is well written -- and so they have very few
recommendations for improvement.
The biggest concern was small effect size, so you might address this
issue further in limitations or discussion.
It might also be good to expand your discussion to talk about other
learning contexts in which remixing might be beneficial. This is an
opportunity to expand and is not necessary.
The reference section needs to be reformatted according to the new
guidelines.
Otherwise, this paper is completely good to go! Congratulations and
thanks for participating in CSCW!
------------------------ Submission 423, Review 1 ------------------------
Title: Remixing as a Pathway to Computational Thinking
Overall rating: 4 (scale is 1..5; 5 is best)
Reviewer expertise: 2 (scale is 1..4; 2 = "Passing Knowledge")
Contribution and Criteria for Evaluation
This paper analyzes log file data from Scratch to show that remix
activity contributes to learning programming constructs.
Expertise
2 (Passing Knowledge)
Assessment of the Paper
As the authors point out, the decision of the Scratch designers to so
strongly support remixing is by no means obvious. As a result, this
analysis of whether remixing is educational valuable is significant.
This paper is well written and careful. I’m generally convinced by its
main point. I’m slightly concerned that the effect size is small.
Papers with a large n and small effect size are worrisome—does this
really mean much? It helps that the authors have gone to great effort to
detail limitations of their work and discuss them clearly.
I’m not familiar with some of the statistical tests used in this work.
It’s important that someone with experience with all these tests check
the paper’s stats.
Requested Revisions
Please remove the word "strong" from the first sentence of the discussion
section. The effect sizes are small, and that isn't really "strong"
support in my view.
Formatting and Reference Issues
First Round Overall Recommendation
4 (Probably acceptable (with minor modifications))
------------------------ Submission 423, Review 2 ------------------------
Title: Remixing as a Pathway to Computational Thinking
Overall rating: 4 (scale is 1..5; 5 is best)
Reviewer expertise: 2 (scale is 1..4; 2 = "Passing Knowledge")
Contribution and Criteria for Evaluation
The intended contribution of this paper is a large-scale quantitative
analysis of whether remixing leads to learning, both broadly (repertoire)
and narrowly (use to CT concepts). To be considered a meaningful
contribution, this analysis should report significant effects of remixing
on both repertoire size and concept usage, and an interpretation of these
results.
Expertise
2 (Passing Knowledge)
Assessment of the Paper
The paper is well-written and organized. The authors do an excellent job
introducing the context and summarizing relevant literature from both
theoretical and empirical perspective. They make a compelling case that a
large-scale quantitative analysis of a Scratch data would complement
earlier qualitative studies that have been conducted in more limited
contexts. In particular, the longitudinal nature of the dataset allows
the researchers to explore the long-term learning effects of remixing
practice.
The data collection and analysis have been conducted with care, for
instance controlling for a number of potential confounds. Examining the
content and metadata of a user's projects over time, and mapping
computational concepts to specific programming constructs, seems like a
promising approach to handling this volume and type of data.
Unfortunately, while the analysis confirms the authors' initial
hypotheses, with statistical significance (as you might expect from a
sample of this size), the effect sizes were quite marginal (as the
authors acknowledge themselves). In addition, I was hopeful for insights
from analyzing the data on a concept-by-concept level, but there was
little gained in comparing and interpreting the different concepts, given
the similar but marginal effect sizes.
Still, the analysis does give initial confirmatory evidence that remixing
is correlated with learning, and contributes methods that could be built
in. For instance, they might be applied in more focused contexts (without
coloring contests), or more formalized educational interventions, where
we larger effect sizes might be observed.
Requested Revisions
More explanation about some the results would be helpful, since some of
them ran against my expectations or intuition. For example, why remixes
have such a strong negative association with particular concepts (much
more negative than the remixes w/ concepts was positive), and why
Downloads is predicted to moderate the effects of remixing.
In the limitations section, the authors discuss copy & pasting as a
shallow demonstration of CT concept in de novo projects. Perhaps they
should include the results of this adjustment in the results section. I
would also be curious to know the precise extent of copy & pasting that
they detected.
There were very few typos that I caught:
Empirical Setting: "(though *___* are more granular)"
Data and Measures: "...we include a control for the number of comments
receive by *an* user"
Limitations: "In line with *out* expectations..."
Discussion: "Although our results *provides* evidence..."
Formatting and Reference Issues
First Round Overall Recommendation
4 (Probably acceptable (with minor modifications))
------------------------ Submission 423, Review 3 ------------------------
Title: Remixing as a Pathway to Computational Thinking
Reviewer: AC-Reviewer
Overall rating: 5 (scale is 1..5; 5 is best)
Reviewer expertise: 3 (scale is 1..4; 3 = "Knowledgeable")
Contribution and Criteria for Evaluation
This paper provides a quantitative examination of whether there is
evidence that remixing is a viable approach to learning programming. The
paper is a well-executed example of using regression models to test
research hypotheses and providing a thoughtful and well-developed
rationale and discussion of the work.
The main criteria for evaluating this type of contribution is both the
quality of the quantitative test, and (for the CSCW audience) the quality
of the rationale and explanation/discussion of results surrounding the
empirical data.
Expertise
3 (Knowledgeable)
Assessment of the Paper
This is a beautifully written and well-presented paper examining whether
the remixing philosophy that underlies the Scratch community's approach
to learning programming is actually effective, in line with the
reputation it enjoys in the social computing literature. The authors look
at a huge dataset of Scratch programming projects, and describe a careful
and well-executed approach to two fundamental hypotheses: 1) that more
remixing is associated with more growth in a learner's repertoire of
Scratch programming blocks, and 2) that a learner who has remixed more
projects containing a particular computational thinking concepts will be
more likely to incorporate those concepts into a future de novo project.
The degree of care and thought put into testing both of these hypotheses
with an extremely large dataset is evident throughout the paper.
Related to this is the excellent writing of the paper, which is
intimately tied to the explanation of the research and its results. The
authors lead readers through their process and thinking step by step,
from cleansing of the data, to sharing the exact models and the rationale
behind various control variables that were added as the exploratory
models were built up, and to the results, and the limitations or cautions
on the results. The authors have a very nice way of connecting their work
thoroughly with prior research ("We are also inspired by work by ...")
and pointing out along the way findings of particular interest, or that
inform in new ways on previously established findings.
I looked in vain for any weaknesses that might undermine confidence in
the paper's claims or results. Finally, I found one!! I think. In the
Results section, paragraph 4, you said you showed plots for two
prototypical users in "Table 4," whereas I think you mean here "Figure
4." There. Don't say R3 didn't contribute anything of value...
Seriously, this is a model paper that gets everything right, that has the
added bonus of making a highly quantitative approach understandable by a
wide range of CSCW readers. I look forward to seeing it in the program.
Requested Revisions
Um, change "Table 4" to "Figure 4."
Formatting and Reference Issues
The references are not in the new format, but since I really dislike the
new format...
First Round Overall Recommendation
5 (Definitely acceptable (ready as-is))