collective/cdsc_examples_repository
17
0
Fork 0
Code

added reviews from scratch community blocks paper (CHI2017)

Browse Source
This commit is contained in:
Benjamin Mako Hill 2020-11-24 12:42:30 -08:00
parent 6a7a781a81
commit 8661aa854a
4 changed files with 758 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
chi_rebuttals/2017-scratch_community_blocks/README.txt Normal file
View File

@ -0,0 +1,7 @@
Rebuttal and other information for:
Dasgupta, Sayamindu, and Benjamin Mako Hill. 2017. “Scratch Community
Blocks: Supporting Children As Data Scientists.” In Proceedings of the
2017 CHI Conference on Human Factors in Computing Systems (CHI ’17),
3620–3631. New York, New York:
ACM. https://doi.org/10.1145/3025453.3025847.

37
chi_rebuttals/2017-scratch_community_blocks/scratch_community_blocks-CHI2017-20161223-rebuttal.txt Normal file
View File

@ -0,0 +1,37 @@
First, we want to thank the reviewers for their helpful feedback! We believe that the changes described below will address nearly all of concerns raised by the reviewers. Although our manuscript is already 10 full pages, we can edit out redundant text and tighten our text. We can also streamline our conclusion to remove more than a paragraph. We will use this extra room to make the adjustments described below.
1.
R1, R2, and R4 each urged us to go beyond isolated case studies to describe how our system was used in general. We will make several additions to the paper to attempt to characterize overall patterns of use.
First, we will describe the relative frequency that individual blocks were used in the user-created projects and we will visualize these data in a histogram. If we cannot create space to include this figure, we will place it in the supplementary material.
Second, to give readers a better sense of how widespread the out-of-context block issue was, we will report the proportion of users (~19%) who used out-of-context blocks.
More specifically, R1 and R4 asked us to characterize and reflect on how our system may have failed to engage some users in ways that might not be reflected in our case-studies. Toward this end, we will briefly report summary statistics describing the relative levels of activity across users in the beta test. As is the case in nearly all informal learning communities (and in Scratch in general), there is a wide distribution of activity levels. A majority of our invitees shared no projects at all.
Given our lack of visibility into users invited to beta test, we cannot know if relatively inactive users attempted but failed to use the system due to design limitations or if they simply were uninterested, distracted, busy, etc. We will mention this in our limitations section. Additionally, we will explain that both our beta test and our survey relied on voluntary participation by users and our attempts to characterize usage in our sample may be unrepresentative of the ways that the system would be used in other contexts, like schools, where participation is required.
2.
R3 expressed concern that our description of Scratch Community Blocks users as “data scientists” may be an overstatement. We will edit our conclusion to make it clear that when we claim that children are data scientists, we do so in a way that seeks to parallel the way that Papert described Logo as means toward, “teaching children to be mathematicians vs. teaching about mathematics.”
We agree with R3 that users of our system are limited in many ways that professional data scientists are not. We call our users data scientists as a way of contrasting our approach — allowing youth to engage in the process of doing data science by writing computer programs to ask and answer their own questions using data — with the alternative approach of simply teaching youth about data science. We will cite Papert’s article in our discussion to clarify this phrasing.
3.
R1 pointed out that the paper did not explicitly describe specific learning goals or outcomes. We will edit the “new pathways...” subsection to describe specific learning goals. Our learning goals include reflection as well as the ability to access, analyze, and visualize of data through programming.
4.
R1 and R4 asked us to reflect on our decision to not include visualization primitives and to describe the trade-offs associated with our approach of using sample projects instead. We will add text to our limitations section to discuss this issue.
One problem we already describe, that we will reframe in terms of this limitation, is the problem users encountered with scaling visualizations which could have been avoided with a primitive. We will also explain that although only two users requested visualization features in our survey, this may be due to our users’ age and relative unfamiliarity with concepts and terminology.
5.
As suggested by R1 and R3, we will edit our background section to add references to studies around how youth interpret and understand visualizations (e.g. Ben-Zvi and Aridor-Berger 2016, Laina and Wilkerson 2016), as well as Shapiro’s work on Blocky Talky.
6.
R1 asked us to report feature requests from users. We will add a line explaining that common requests included more support material, and the system to be faster and more responsive.
7.
We will fix all the typos and grammatical errors pointed out by the reviewers. We will have our manuscript professionally proof-read.
8.
There is also one excellent suggestion that we are not able to make. R1 suggested that we might report changes in the activity of users as a measure of how well the system succeeded in promoting reflection. Including this new analysis is a larger change that we felt we could justify promising in a rebuttal. That said, we are hoping to do this in a future study and we will say as much in our conclusion.

342
chi_rebuttals/2017-scratch_community_blocks/scratch_community_blocks-CHI2017-review.txt Normal file
View File

@ -0,0 +1,342 @@
CHI 2017 Papers and Notes
Reviews of submission #2961: "Scratch Community Blocks: Supporting
Children as Data Scientists"
------------------------ Submission 2961, Review 4 ------------------------
Reviewer: primary (1AC)
Expertise
3 (Knowledgeable)
Recommendation
. . . Between possibly accept and strong accept; 4.5
Award Nomination
If accepted, this paper would be among the top 20% of papers presented at CHI (Best Paper: Honorable Mention nomination)
Your Assessment of this Paper's Contribution to HCI
This well-written paper explores an extension to the Scratch visual
programming system (Scratch Community Blocks), a sub-component that
enables children in the Scratch community to incorporate data analysis
into their programs.
The work contributes to our understanding of how to help young learners
to experiment with data science/analytics 'authentically' -- that is,
using their own community data, in ways that are similar to (or at least
aspire to) the practices of professional data scientists. The design of
the system itself is novel and thoughtful, and the paper includes a
fairly extensive user study/beta test that highlights how children will
likely use the system and learn from it as they use it "in the wild," on
their own personal projects.
The Review
1AC: The Meta-Review
Thank you for your submission to CHI 2017.
All of the reviewers noted that this is a well-written, thoughtful, and
thorough paper about the design and initial user evaluation of an
extension to the Scratch visual programming system (Scratch Community
Blocks). The system design and evaluation are highly relevant to the CHI
community, particularly those involved in the areas of child-computer
interaction and learning.
The introduction and related work sections do an excellent job of
motivating the work (R2), and convincingly connect the need for such a
system to learning theory (R1, R2). As R2 states: "The project
contributes to our understanding of how to empower young learners to be
data scientists and remains faithful to the core constructionist
principles that helped make Scratch such a success."
In addition, R3 noted that the design and goals of the Blocks extension
is very well thought out and uniquely balanced: "This is a tricky design
space where it's important to strike the right balance between
expressiveness and simplicity."
All of the reviewers mentioned that the system description was very
clear, with nice examples of how the extension could be used by children
(or any members of the Scratch community).
While all of the reviewers agree that the submission is detailed and
well-presented, each of them also offers some excellent suggestions to
improve it.
Specifically, all of the reviewers agree that the limitations of the
paper lie in a *missed* opportunity to offer a more nuanced discussion of
how the *entire* community of beta-testers used the extension, beyond the
few examples that were showcased. For example, R2 notes that "we know
little about the typicality of the examples given: the examples provided
are rich; however, the types and range of users who incorporated
Community Blocks -- and what they learned through their efforts --
remains vague." Similarly, R1 observed that "While a hand-picked set of
examples show that the current approach is 'sufficient' to create
innovative, constructionist designs that support self-reflection, we have
no sense of how many of the designs did so, or of the (likely) numerous
cases where users did not succeed at these goals.
R3 also noted that the paper's contribution to data science was somewhat
overstated, given that the Blocks extension included a limited number of
data attributes; its analysis opportunities are limited to Scratch
community data only, and the authors relied mostly on Scratch 'proper' to
create visualizations (rather than including some entry-level
visualization primitives within the extension). Relatedly, R1 would have
liked more emphasis on the ways in which the design promotes (or detracts
from) how young learners learn to code.
Overall, however, the paper remains a strong submission, with a clear
contribution to the HCI (and learning sciences) community.
Rebuttal response
------------------------ Submission 2961, Review 1 ------------------------
Reviewer: secondary (2AC)
Expertise
4 (Expert )
Recommendation
. . . Between possibly accept and strong accept; 4.5
Award Nomination
If accepted, this paper would be among the top 20% of papers presented at CHI (Best Paper: Honorable Mention nomination)
Your Assessment of this Paper's Contribution to HCI
This paper describes how the Scratch visual programming language was
modified to allow beta testers to incorporate data analytics (about the
Scratch community and its members) into their programs. Specifically, it
describes how the design approaches they used instantiated the
constructionist goals of improving education through support of creation
and self-reflection. Several examples that illustrate the successes of
their design are shared, as well as some limitations of their design.
The Review
Overall, this was a clearly thought out and presented paper that does a
nice job of tying constructionist educational theory to specific designs.
The design itself, which allowed Scratch users to create programs that
integrated data from the Scratch community, was well thought-out and
carefully explained, along with how it connected to theory. It also was
unique, in its focus on supporting data analytics in a programming
context, its focus on youth (who do not typically analyze data like
this), and its focus on supporting personally-relevant data.
The examples that were provided did an excellent job of showing the
potential of this approach. I was particularly interested in the example
of using community data for other purposes - i.e., in a game. While that
deviated from the main framing of the work (on "data analytics"), it
showed how players integrated community data into the Scratch paradigm.
While I recommend the paper be accepted based on those significant
contributions, I believe there is room for improvement. A few specific
issues are raised below:
1) It wasn't ever clear what the youth were supposed to be learning. Were
there any desirable learning outcomes for them? Even constructionist
learning has some end-goal in mind. While you may not have had specific
learning outcomes identified ahead of time, it would be nice for the
authors to address this issue and more clearly articulate what learning
they hoped this new functionality would engender. Currently, it seems
quite vague.
2) The authors could do a better job with the limitations section.
Currently, it focuses on a fairly narrow problem (i.e., a common
"syntactical" mistake that was made). However, much more could be said
about the potential downsides of the current approach. While a
hand-picked set of examples show that the current approach is
"sufficient" to create innovative, constructionist designs that support
self-reflection, we have no sense of how many of the designs did so, or
of the (likely) numerous cases where users did not succeed at these
goals. There are clear costs to the current approach of, say, not
including data visualizations. Did you see evidence of users wanting
those or asking for them? What did your interviews and class say about
desired improvements? Is there any ways that the current designs failed
to promote self-reflection for some users?
3) Overall, the literature review was fleshed out fairly well. One area
it didn't delve into, which seems highly relevant, is the literature that
examines the process that youth use to analyze and visualize or make
sense of data. What are the difficult steps in the process? What mistakes
do they make? How do tools help them? Including some of the findings on
"how" people make sense of data would enrich the current paper and help
clarify how this process might be different in the current design than it
would be with other more traditional designs (e.g., standard analysis and
visualization tools). Having said that, there is a lot of research on how
students develop "statistical thinking" and "data modeling" skills, but
not necessarily within a programming paradigm. The authors may want to
consider this suggestion, but at the same time, leave a full analysis for
this to another paper.
Below are some minor suggestions or errors:
You may want to mention leading visualization and analysis tools such as
Tableau and R in the paragraph starting “In addition to these systems,
educators have frequently…”
Typos and grammatical errors:
“The design of… are informed” – should be “is” instead of
“are”
“an unique” should be “a unique”
“is built around a the idea”
“in ways that prompts”
“Although [the] Scratch language itself”
“Scratch can [be] described as”
“on [the] two most salient”
“site for interaction [and] are the second most”
“As a result[,] usernames are”
“interviews transcripts”
“representing the the”
“are more 1.2 times more”
Rebuttal response
------------------------ Submission 2961, Review 2 ------------------------
Reviewer: external
Expertise
4 (Expert )
Recommendation
. . . Between possibly accept and strong accept; 4.5
Award Nomination
If accepted, this paper would be among the top 20% of papers presented at CHI (Best Paper: Honorable Mention nomination)
Your Assessment of this Paper's Contribution to HCI
The contribution of this work is providing a compelling and successful
example of letting young learners be data scientists. By leveraging both
the interface affordances and the social meaningfulness of Scratch, the
author(s) empower learners to ask and answer questions about this popular
learning environment. In doing so, the authors of this extension to
Scratch have created a way for learners to engage in authentic data
science practices.
The Review
The paper presents the Scratch Community Blocks extension to the Scratch
programming platform. These blocks enable users to gather data about
other users and projects in the Scratch universe and, by giving access to
that data through Scratch commands directly, allows the user to use that
within their own Scratch projects. The project contributes to our
understanding of how to empower young learners to be data scientists and
remains faithful to the core constructionist principles that helped make
Scratch such a success.
The paper does a nice job of motivating the work and presenting relevant
related research. The description of the design and implementation is
sufficient to understand the system and what the contribution of the work
to the Scratch environment is. The samples programs (Figure 4) were
helpful for being able to understand what it looks like to use the blocks
to answer real questions.
The illustrative examples were both a strength and weakness of the paper.
Strength in that they provide very compelling examples of what it means
for learners to be data scientists and proved the effectiveness of the
design. The visualization made with ice cream scoops serves as a
wonderful example of the creativity and fun that is possible by
empowering learners with compelling tools and learning contexts. The
examples were a weakness of the paper in that I want to know way more
about each example as well as what happened with other users of the
system. My concern here is that we know little about the typicality of
the examples given. Are Community Blocks only
useful/accessible/meaningful for the top 1% of Scratchers? Were the
issues mentioned in limitations section (in particular the
context-sensitive accessor blocks) a serious hurdle for learners? In the
next version of the paper, I hope to see more text dedicated to this
portion of the paper.
Rebuttal response
------------------------ Submission 2961, Review 3 ------------------------
Reviewer: external
Expertise
4 (Expert )
Recommendation
Strong Accept: I would argue strongly for accepting this paper; 5.0
Award Nomination
If accepted, this paper would be among the top 5% of papers presented at CHI (Best Paper nomination)
Your Assessment of this Paper's Contribution to HCI
This is a well-written paper that explores an extension to the Scratch
programming system. The big picture idea is to put data science tools
into the hands of kids by creating simple scratch blocks that connect to
the larger scratch community database. So, for example, you can write a
simple program to count the number of users from a certain country that
follow you. This design was backed up by a fairly extensive user study
and analysis of the types of projects that kids are creating with these
tools in the wild.
The Review
First off, the blocks (language) themselves seem very well thought out.
This is a tricky design space where it's important to strike the right
balance between expressiveness and simplicity. Too complicated and kids
won't use the blocks; not expressive enough and kids won't be able to do
anything interesting. Backing up this design with real users in the wild
is impressive. In this regard, the Illustrative Examples section does a
good job explaining the range of features the new blocks enable.
An evaluation was conducted with 721 users who created 1,659 projects
with the new blocks. This was supplemented with in-person workshops and
interviews from 12 local users. The authors do a nice job detailing the
creative potential of the blocks in the hands of real users. The example
projects selected highlight some of the ways in which kids can create
personally relevant visualizations of scratch community data.
While the blocks create interesting new ties to the scratch community,
I'm not entirely convinced by the data science argument. The users
highlighted did, in some cases, create visualizations of different
aspects of the scratch community. That said, there are a number of
limitations that would severely restrict the data science that could be
done: 1) the blocks are limited in terms of data scope. I can't use these
blocks to visualize or explore other data sets beyond the scratch
community. And even within the scratch community, I can only visualize a
handful of data attributes; 2) there aren't any visualization blocks.
This was an intentional design choice because kids can use scratch itself
to create visualizations. But it might still be nice to have some entry
level visualization primitives to jumpstart student work; and 3) I can
only work with "flat" data sets of a lists of single values. This isn't
really a criticism of the work itself--this is an incredibly difficult
design space to work in. I just think the framing could be a little less
aggressive on the data science angle.
The paper ends with some nice reflection on design tradeoffs around core
scratch design principles, data scope, and the advantages of things like
syntax errors.
It might be worth mentioning Ben Shapiro's work with Blockly Talky.
Rebuttal response

372
chi_rebuttals/2017-scratch_community_blocks/scratch_community_blocks-CHI2017-review_postpc.txt Normal file
View File

@ -0,0 +1,372 @@
CHI 2017 Papers and Notes
Reviews of submission #2961: "Scratch Community Blocks: Supporting
Children as Data Scientists"
------------------------ Submission 2961, Review 4 ------------------------
Reviewer: primary (1AC)
Overall rating: 4.5 (scale is 0.5..5; 5 is best)
Expertise
3 (Knowledgeable)
Recommendation
. . . Between possibly accept and strong accept; 4.5
Award Nomination
If accepted, this paper would be among the top 20% of papers presented at CHI (Best Paper: Honorable Mention nomination)
Your Assessment of this Paper's Contribution to HCI
This well-written paper explores an extension to the Scratch visual
programming system (Scratch Community Blocks), a sub-component that
enables children in the Scratch community to incorporate data analysis
into their programs.
The work contributes to our understanding of how to help young learners
to experiment with data science/analytics 'authentically' -- that is,
using their own community data, in ways that are similar to (or at least
aspire to) the practices of professional data scientists. The design of
the system itself is novel and thoughtful, and the paper includes a
fairly extensive user study/beta test that highlights how children will
likely use the system and learn from it as they use it "in the wild," on
their own personal projects.
The Review
1AC: The Meta-Review
--------------------------------------------------------
POST-REBUTTAL and POST-PC MEETING COMMENTS
--------------------------------------------------------
Thank you, authors, for your submission to CHI 2017, and for your
thoughtful and thorough rebuttal.
**Congratulations to the authors on their paper’s conditional
acceptance.**
To gain final approval for acceptance, the authors should address the
reviewer comments, and implement the changes as requested by the
reviewers.
The deadline for PCS submission for consideration for final acceptance is
6 January 2017 (20:00 EST). Congratulations again!
------------------------------------------------------
All of the reviewers noted that this is a well-written, thoughtful, and
thorough paper about the design and initial user evaluation of an
extension to the Scratch visual programming system (Scratch Community
Blocks). The system design and evaluation are highly relevant to the CHI
community, particularly those involved in the areas of child-computer
interaction and learning.
The introduction and related work sections do an excellent job of
motivating the work (R2), and convincingly connect the need for such a
system to learning theory (R1, R2). As R2 states: "The project
contributes to our understanding of how to empower young learners to be
data scientists and remains faithful to the core constructionist
principles that helped make Scratch such a success."
In addition, R3 noted that the design and goals of the Blocks extension
is very well thought out and uniquely balanced: "This is a tricky design
space where it's important to strike the right balance between
expressiveness and simplicity."
All of the reviewers mentioned that the system description was very
clear, with nice examples of how the extension could be used by children
(or any members of the Scratch community).
While all of the reviewers agree that the submission is detailed and
well-presented, each of them also offers some excellent suggestions to
improve it.
Specifically, all of the reviewers agree that the limitations of the
paper lie in a *missed* opportunity to offer a more nuanced discussion of
how the *entire* community of beta-testers used the extension, beyond the
few examples that were showcased. For example, R2 notes that "we know
little about the typicality of the examples given: the examples provided
are rich; however, the types and range of users who incorporated
Community Blocks -- and what they learned through their efforts --
remains vague." Similarly, R1 observed that "While a hand-picked set of
examples show that the current approach is 'sufficient' to create
innovative, constructionist designs that support self-reflection, we have
no sense of how many of the designs did so, or of the (likely) numerous
cases where users did not succeed at these goals.
R3 also noted that the paper's contribution to data science was somewhat
overstated, given that the Blocks extension included a limited number of
data attributes; its analysis opportunities are limited to Scratch
community data only, and the authors relied mostly on Scratch 'proper' to
create visualizations (rather than including some entry-level
visualization primitives within the extension). Relatedly, R1 would have
liked more emphasis on the ways in which the design promotes (or detracts
from) how young learners learn to code.
Overall, however, the paper remains a strong submission, with a clear
contribution to the HCI (and learning sciences) community.
Rebuttal response
------------------------ Submission 2961, Review 1 ------------------------
Reviewer: secondary (2AC)
Overall rating: 4.5 (scale is 0.5..5; 5 is best)
Expertise
4 (Expert )
Recommendation
. . . Between possibly accept and strong accept; 4.5
Award Nomination
If accepted, this paper would be among the top 20% of papers presented at CHI (Best Paper: Honorable Mention nomination)
Your Assessment of this Paper's Contribution to HCI
This paper describes how the Scratch visual programming language was
modified to allow beta testers to incorporate data analytics (about the
Scratch community and its members) into their programs. Specifically, it
describes how the design approaches they used instantiated the
constructionist goals of improving education through support of creation
and self-reflection. Several examples that illustrate the successes of
their design are shared, as well as some limitations of their design.
The Review
Overall, this was a clearly thought out and presented paper that does a
nice job of tying constructionist educational theory to specific designs.
The design itself, which allowed Scratch users to create programs that
integrated data from the Scratch community, was well thought-out and
carefully explained, along with how it connected to theory. It also was
unique, in its focus on supporting data analytics in a programming
context, its focus on youth (who do not typically analyze data like
this), and its focus on supporting personally-relevant data.
The examples that were provided did an excellent job of showing the
potential of this approach. I was particularly interested in the example
of using community data for other purposes - i.e., in a game. While that
deviated from the main framing of the work (on "data analytics"), it
showed how players integrated community data into the Scratch paradigm.
While I recommend the paper be accepted based on those significant
contributions, I believe there is room for improvement. A few specific
issues are raised below:
1) It wasn't ever clear what the youth were supposed to be learning. Were
there any desirable learning outcomes for them? Even constructionist
learning has some end-goal in mind. While you may not have had specific
learning outcomes identified ahead of time, it would be nice for the
authors to address this issue and more clearly articulate what learning
they hoped this new functionality would engender. Currently, it seems
quite vague.
2) The authors could do a better job with the limitations section.
Currently, it focuses on a fairly narrow problem (i.e., a common
"syntactical" mistake that was made). However, much more could be said
about the potential downsides of the current approach. While a
hand-picked set of examples show that the current approach is
"sufficient" to create innovative, constructionist designs that support
self-reflection, we have no sense of how many of the designs did so, or
of the (likely) numerous cases where users did not succeed at these
goals. There are clear costs to the current approach of, say, not
including data visualizations. Did you see evidence of users wanting
those or asking for them? What did your interviews and class say about
desired improvements? Is there any ways that the current designs failed
to promote self-reflection for some users?
3) Overall, the literature review was fleshed out fairly well. One area
it didn't delve into, which seems highly relevant, is the literature that
examines the process that youth use to analyze and visualize or make
sense of data. What are the difficult steps in the process? What mistakes
do they make? How do tools help them? Including some of the findings on
"how" people make sense of data would enrich the current paper and help
clarify how this process might be different in the current design than it
would be with other more traditional designs (e.g., standard analysis and
visualization tools). Having said that, there is a lot of research on how
students develop "statistical thinking" and "data modeling" skills, but
not necessarily within a programming paradigm. The authors may want to
consider this suggestion, but at the same time, leave a full analysis for
this to another paper.
Below are some minor suggestions or errors:
You may want to mention leading visualization and analysis tools such as
Tableau and R in the paragraph starting “In addition to these systems,
educators have frequently…”
Typos and grammatical errors:
“The design of… are informed” – should be “is” instead of
“are”
“an unique” should be “a unique”
“is built around a the idea”
“in ways that prompts”
“Although [the] Scratch language itself”
“Scratch can [be] described as”
“on [the] two most salient”
“site for interaction [and] are the second most”
“As a result[,] usernames are”
“interviews transcripts”
“representing the the”
“are more 1.2 times more”
1AC: The Meta-Review
Rebuttal response
I have read the rebuttal and am pleased with the responses. While I have
not changed my score, I believe the 4.5 score is a good indication of the
paper's value and recommend it be accepted.
------------------------ Submission 2961, Review 2 ------------------------
Reviewer: external
Overall rating: 4.5 (scale is 0.5..5; 5 is best)
Expertise
4 (Expert )
Recommendation
. . . Between possibly accept and strong accept; 4.5
Award Nomination
If accepted, this paper would be among the top 20% of papers presented at CHI (Best Paper: Honorable Mention nomination)
Your Assessment of this Paper's Contribution to HCI
The contribution of this work is providing a compelling and successful
example of letting young learners be data scientists. By leveraging both
the interface affordances and the social meaningfulness of Scratch, the
author(s) empower learners to ask and answer questions about this popular
learning environment. In doing so, the authors of this extension to
Scratch have created a way for learners to engage in authentic data
science practices.
The Review
The paper presents the Scratch Community Blocks extension to the Scratch
programming platform. These blocks enable users to gather data about
other users and projects in the Scratch universe and, by giving access to
that data through Scratch commands directly, allows the user to use that
within their own Scratch projects. The project contributes to our
understanding of how to empower young learners to be data scientists and
remains faithful to the core constructionist principles that helped make
Scratch such a success.
The paper does a nice job of motivating the work and presenting relevant
related research. The description of the design and implementation is
sufficient to understand the system and what the contribution of the work
to the Scratch environment is. The samples programs (Figure 4) were
helpful for being able to understand what it looks like to use the blocks
to answer real questions.
The illustrative examples were both a strength and weakness of the paper.
Strength in that they provide very compelling examples of what it means
for learners to be data scientists and proved the effectiveness of the
design. The visualization made with ice cream scoops serves as a
wonderful example of the creativity and fun that is possible by
empowering learners with compelling tools and learning contexts. The
examples were a weakness of the paper in that I want to know way more
about each example as well as what happened with other users of the
system. My concern here is that we know little about the typicality of
the examples given. Are Community Blocks only
useful/accessible/meaningful for the top 1% of Scratchers? Were the
issues mentioned in limitations section (in particular the
context-sensitive accessor blocks) a serious hurdle for learners? In the
next version of the paper, I hope to see more text dedicated to this
portion of the paper.
Rebuttal response
------------------------ Submission 2961, Review 3 ------------------------
Reviewer: external
Overall rating: 5 (scale is 0.5..5; 5 is best)
Expertise
4 (Expert )
Recommendation
Strong Accept: I would argue strongly for accepting this paper; 5.0
Award Nomination
If accepted, this paper would be among the top 5% of papers presented at CHI (Best Paper nomination)
Your Assessment of this Paper's Contribution to HCI
This is a well-written paper that explores an extension to the Scratch
programming system. The big picture idea is to put data science tools
into the hands of kids by creating simple scratch blocks that connect to
the larger scratch community database. So, for example, you can write a
simple program to count the number of users from a certain country that
follow you. This design was backed up by a fairly extensive user study
and analysis of the types of projects that kids are creating with these
tools in the wild.
The Review
First off, the blocks (language) themselves seem very well thought out.
This is a tricky design space where it's important to strike the right
balance between expressiveness and simplicity. Too complicated and kids
won't use the blocks; not expressive enough and kids won't be able to do
anything interesting. Backing up this design with real users in the wild
is impressive. In this regard, the Illustrative Examples section does a
good job explaining the range of features the new blocks enable.
An evaluation was conducted with 721 users who created 1,659 projects
with the new blocks. This was supplemented with in-person workshops and
interviews from 12 local users. The authors do a nice job detailing the
creative potential of the blocks in the hands of real users. The example
projects selected highlight some of the ways in which kids can create
personally relevant visualizations of scratch community data.
While the blocks create interesting new ties to the scratch community,
I'm not entirely convinced by the data science argument. The users
highlighted did, in some cases, create visualizations of different
aspects of the scratch community. That said, there are a number of
limitations that would severely restrict the data science that could be
done: 1) the blocks are limited in terms of data scope. I can't use these
blocks to visualize or explore other data sets beyond the scratch
community. And even within the scratch community, I can only visualize a
handful of data attributes; 2) there aren't any visualization blocks.
This was an intentional design choice because kids can use scratch itself
to create visualizations. But it might still be nice to have some entry
level visualization primitives to jumpstart student work; and 3) I can
only work with "flat" data sets of a lists of single values. This isn't
really a criticism of the work itself--this is an incredibly difficult
design space to work in. I just think the framing could be a little less
aggressive on the data science angle.
The paper ends with some nice reflection on design tradeoffs around core
scratch design principles, data scope, and the advantages of things like
syntax errors.
It might be worth mentioning Ben Shapiro's work with Blockly Talky.
Rebuttal response
I have read the authors' rebuttal and thank them for taking the reviewer
comments into account.