collective/cdsc_examples_repository
18
0
Fork 0
Code

added material for hautea et al. CHI 2017 paper

Browse Source
This commit is contained in:
Benjamin Mako Hill 2020-11-24 12:39:54 -08:00
parent 4282b22849
commit 6a7a781a81
5 changed files with 900 additions and 0 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

7
chi_rebuttals/2017-critical_data_literacies/README.txt Normal file
View File

@ -0,0 +1,7 @@
Rebuttal for:
Hautea, Samantha, Sayamindu Dasgupta, and Benjamin Mako
Hill. 2017. “Youth Perspectives on Critical Data Literacies.” In
Proceedings of the 2017 CHI Conference on Human Factors in Computing
Systems (CHI ’17), 919–930. New York, New York:
ACM. https://doi.org/10.1145/3025453.3025823.

34
chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-rebuttal-CHI2017.txt Normal file
View File

@ -0,0 +1,34 @@
We thank the reviewers for their careful attention and feedback. Below, we describe adjustments we plan to make that we believe will address the reviewers central concerns. Although relatively minor, we agree that these changes will improve the manuscript enormously. We are confident that we can tighten our text and remove discussion of less relevant concepts (as described below) to accommodate the proposed additions.
1.
R1 and R4 suggest that our work is unfocused and discusses too many loaded terms: data literacies, data fluencies, social media, big data, data analytics, and data science. As per R1, we will revise our paper so that its contribution is more narrowly framed in terms of “critical data fluencies.” As per R2 and R3, we will return to this concept in the discussion and refer to it when presenting our findings to offer a more cohesive narrative.
As per R1 and R4, we will remove discussion of social media and data science. As per R1, we will de-emphasize discussion of big data except where studies of big data complement our findings in ways that were cited by R2 as central to our work’s contribution (e.g. connections to boyd and Crawford).
Given these changes, we feel that our previous title is inappropriate. We will retitle the manuscript “Youth Perspectives of Critical Data Fluencies.”
2.
A concern given this reframing is R2’s point that we were inconsistent in our use of the terms literacies and fluencies. We will standardize on the term fluencies and add citations to work that advocates for this term over literacies (e.g. Resnick, Rusk, & Cooke, 1998).
3.
As per R1 and R4, we will clarify our methodology. We will explain that our approach was drawn directly from Charmaz (2006)’s textbook on grounded theory (GT). As per Charmaz, we iteratively produced a single codebook with various sources of data as a means of triangulation. We will explain that although we coded all data, we coded “earlier” data more openly and heavily and coded later data (including the survey) more lightly and with more reliance on existing codes. Our QDA software (Dedoose) allowed us to code interview transcripts, images, and field notes in the same interface.
R1 and R4 asked for details about our codes. Although we have not seen it done, we will export the full list of codes we have used and publish it as supplementary material if requested.
4.
As per R4, we will explain how we used deductive codes. Although some early descriptions of GT advocate coding with no preconceived concepts, we followed Charmaz’s call for “an open mind, not an empty head” (pg. 174) and the use of both “sensitizing” codes drawn from theory during initial coding as well as later-stage iterative recoding of data using codes informed by theory relevant to emergent themes. We will make this process explicit in our methodology section.
5.
R3 and R4 suggested that our introduction should clearly outline the rest of our paper. We will edit our introduction to do this.
6.
R1 and R4 asked us to reflect on how our findings might generalize and how broadly representative we believe the concerns raised by kids in our sample are. We will explain that although we believe that our themes are a valid description of the major ways that kids discussed the social implications of data in our sample, they reflect the aggregate observations of active and engaged Scratch users. We will explain that no child raised every issue and that it is possible that a system with different affordances might surface different concerns. Moreover, we will remind readers that open questions around generalizability are a frequently-cited limitation of GT.
7.
In a related sense, R1, R2, and R4 asked us whether participants translated their insights from Scratch to other social media platforms. We will add text to our discussion to explain that, although there are similarities between Scratch and other systems that might support this, we have little evidence that this occurred. We considered asking questions about this on our survey but did not because we felt that doing so would involve collecting evidence that users under 13 had broken social media websites’ ToS which usually disallows these users. We will raise this as a limitation of our study.
8.
R1 requested that we include more quotes to address concerns about cherry-picking. As per Charmaz, we selected quotes that we felt were clear reflections of the themes we identified. Where possible — including the “creepy but cool” example — we will include additional quotes. That said, we are limited by space and even with improvement in this regard, our paper will only showcase a tiny amount of our data. We hope that several additional quotes and more detailed description of our methodology can alleviate R1’s concerns.
9.
We will fix all the typos and grammatical errors pointed out by the reviewers and have our manuscript professionally proof-read.

444
chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017-postpc.txt Normal file
View File

@ -0,0 +1,444 @@
CHI 2017 Papers and Notes
Reviews of submission #2826: "“Creepy and Very Cool”: Youth
Perspectives on Social Media Data Analytics"
------------------------ Submission 2826, Review 4 ------------------------
Reviewer: primary (1AC)
Overall rating: 4 (scale is 0.5..5; 5 is best)
Expertise
2 (Passing Knowledge)
Recommendation
Possibly Accept: I would argue for accepting this paper; 4.0
Award Nomination
If accepted, this paper would not be among the top 20% of papers presented at CHI
Your Assessment of this Paper's Contribution to HCI
This paper reports a qualitative study of youth interaction with social
media usage data. In particular participants used an extension of the
Scratch programming environment, which allows them to write programs to
process data about other participants’ behavior on such extended
Scratch system. This is somewhat similar to the kind of data available on
other social networks.
The data, analyzed through grounded theory, illustrates how youths make
sense of the data and its implications on privacy.
The Review
1AC: The Meta-Review
All reviewers are generally positive about the contribution made by this
paper, as well its timeliness and novelty. R2 is particularly
enthusiastic about the paper, while the other two reviewers flag some
concerns. While all reviewers agree that the paper is generally well
written, they all suggest some re-framing, with R1 and R3 being
particularly critical about this matter.
In particular, all reviewers suggest that the paper would benefit from
reducing the emphasis on big data, and instead narrowing the focus of the
introduction. R1 offers a very practical suggestion, to focus on
“learning about one’s data is the most interesting”. This is the
key issue that the authors need to address in their rebuttal.
Other issues that would be good to see addressed in the rebuttal include:
- clarification of the analysis; more information should be provided
about the codes used for different parts of the data (e.g. interviews vs.
survey responses), and how these were combined (R1); moreover, it would
be good to see a justification of the use of “deductive codes drawn
from existing theory” within a grounded theory approach (which is
normally focused just on inductive codes emerging from the data) (AC1).
- The introduction should more clearly outline the rest of the paper, to
prepare readers for what to expect (R3)
- discussion of how general or specific the reported findings are (R1)
- more information about whether participants translated some of their
insight from Scratch to other social media platforms (R1, R2)
*** Comments after rebuttal and PC meeting ***
Congratulations to the authors on their paper’s conditional acceptance.
To gain final approval for acceptance, the authors should address the
reviewer comments and implement the changes as proposed in the rebuttal.
The deadline for PCS submission for consideration for final acceptance is
6 January 2017 (20:00 EST).
Rebuttal response
The rebuttal convinced me that the authors can address the issues raised
by reviewers and improve the paper to a level suitable for publication at
CHI. In light of the rebuttal, I believe the paper can make a timely and
important contribution at CHI, so I raise my score from 3.5 to 4.
The external reviewers also received positively the rebuttal, and two of
them raised their scores.
------------------------ Submission 2826, Review 5 ------------------------
Reviewer: secondary (2AC)
Overall rating: 4 (scale is 0.5..5; 5 is best)
Expertise
3 (Knowledgeable)
Recommendation
Possibly Accept: I would argue for accepting this paper; 4.0
Award Nomination
Your Assessment of this Paper's Contribution to HCI
The Review
This paper addresses a timely and important issue. The reviewers are in
agreement that there is much value in this work. In order to move
forward, I would recommend that the authors focus on two main issues
present in the reviewers' comments in their rebuttal.
First, all reviewers comment on the confusion regarding the framing of
the work, which has the outcome of obscuring the main contributions of
the research. The intended focus should be clarified.
Second, the methods used should be further detailed. Particularly,
answering the questions asked by R1 would go a long way towards
convincing readers that the work done in this project is reliable and
valid.
1AC: The Meta-Review
Rebuttal response
------------------------ Submission 2826, Review 1 ------------------------
Reviewer: external
Overall rating: 4 (scale is 0.5..5; 5 is best)
Expertise
4 (Expert )
Recommendation
Possibly Accept: I would argue for accepting this paper; 4.0
Award Nomination
Your Assessment of this Paper's Contribution to HCI
This paper contributes into how young people think about and reflect on
their own data in the context of Scratch. HCI researchers could use this
to better design technologies for young people, and to think about
privacy and related theoretical issues related to youth and data.
The Review
The premise of this paper is promising. I agree youth perspectives on big
data are understudied and I’m very excited to see efforts to explore
their perspectives. I find the choice of using Scratch to study youths’
perspectives an intriguing one. The main question I have is that Scratch
is an educational platform, hosted by MIT, and it does not advertise or
sell data to anyone (as far as I know). This puts it in a dramatically
different category of platforms compared to stereotypical big data sites
like Facebook or Twitter or Google or even educational sites like MOOCs
or Canvas, which are ultimately for profit industries. How would the
authors frame the contributions here given that big data concerns have
largely been about these other categories? (My personal opinion is that
the big data framing actually does a disservice to the narrative by
distracting the reader. This is about how youth learn about and make
sense of data that is collected and displayed about themselves).
The authors collected so much data, I would have liked to see more of it
to show evidence of their claims. For example, the statement “we
observed how Scratch users are grasping the implications of what it means
to have this data out there” – I wanted to know what are they
grasping at? How are they grasping? We are given one “creepy but
cool” quote which is also the title of the paper, and it feels
cherry-picked without more data to support the claim.
I would have liked to see a more systematic and in-depth analysis of the
survey data. 400 responses is a lot and I think it’s worth developing a
codebook and coding the data for key themes then telling the reader about
those. As it is now, the concern about cherry picking data remains
salient because the description of the data coding is hard to discern.
Did the author code all of the data they collected? (interviews, survey,
log files, workshop notes) It seems like it would be very hard to develop
a single codebook for all of these different data sources. And either
way, can the authors tell us about what kinds of codes were in the
codebook, how often key codes showed up, etc?
How representative do the authors think the data they present in this
paper is of youths’ attitudes towards big data generally? There’s
some concern about biases here. The authors isolated the most active
Scratch users, then sample from the more engaged among them (presumably,
those who agree to do a survey are particularly active Scratch users).
What would the authors say we learned that can be generalized versus
cannot?
To return to the “big data” framing critique, it seems like the
authors themselves aren’t exactly sure what it is they’re trying to
articulate sometimes. They refer to data literacy, social media data
analytics, data science. These terms are all kind of loaded and they’re
used in various ways throughout the paper. I would encourage the authors
to pick a clearer narrative and terminology (I think the learning about
one’s data is the most interesting—“data science”, “big
data”, and “social media” aren’t really great fits for this
study).
In general, I’m excited by the ideas and motivation of this work. I
would be more positive if the data analysis process were more transparent
and clear throughout, and if the authors could make a more focused
contribution without relying on (what I would argue are) overused popular
terms.
Rebuttal response
I have read the rebuttal. I appreciate the authors' explanation of their
data analysis and would expect to see that and hopefully more details in
the published paper. We don't need to see all of your codes, but we do
need enough to understand what you did and that there was a rigorous and
systematic process followed. I think whether you add some example codes
to the extended methods detail or add them in an appendix is up to you.
I personally don't like the "critical data fluencies" framing as the
predominant framing. It's hard to parse and may not be meaningful to a
broad audience of readers (I don't find references to it in a quick
Google Scholar search). I realize previously I said the framings were too
broad and now I'm saying it's not broad enough. Nonetheless, I would
encourage the authors to take some time to rethink what they want their
overarching contribution and framing to be. The major point in our
reviews which the authors grokked is to figure out what that story is and
make it clear throughout. I've changed my review from a 3.5 to a 4.
------------------------ Submission 2826, Review 2 ------------------------
Reviewer: external
Overall rating: 5 (scale is 0.5..5; 5 is best)
Expertise
3 (Knowledgeable)
Recommendation
Strong Accept: I would argue strongly for accepting this paper; 5.0
Award Nomination
If accepted, this paper would be among the top 5% of papers presented at CHI (Best Paper nomination)
Your Assessment of this Paper's Contribution to HCI
This paper looks at youth who used Scratch "blocks" to access metadata on
other users. The programs they created led the users to think deeply
about issues with Big Data. This study provides an understanding of 1)
youth experiences of Big Data (which has largely been understudied) 2) an
exploration of a system where users both produce and analyze metadata 3)
pedagogical suggestions for teaching data science.
The Review
What I liked:
- There was a lot to like about this paper! It was surprising to me that
the issues with Big Data that scholars have noted were so readily
identified by youth on Scratch Community Blocks. The insights about how
children and teens view issues of Big Data were in line with things that
researchers have noted in their own critiques, but some of these issues
were presented in a new light when viewed through youths' perspectives.
- I was initially skeptical about how such simple changes to Scratch
could provoke so much reflection from users, but I was thoroughly
convinced by the end of the paper that even relatively small changes to
Scratch had a significant impact on how users understood their
relationship to their data and the ethics of algorithms.
- I was impressed with the rather comprehensive methods used by the
researchers.
- The authors made strong arguments for using these sorts of techniques
for teaching about the ethics of data science.
Issues:
- I was not clear on the importance of discussing "critical data fluency"
because it was not returned to explicitly later in the paper. The authors
seemed to position this concept as a contribution to discussions about
literacy, so I expected to see more about it.
- The authors say "we focus on what can be termed as critical data
literacies" and then say that "the themes that we find are better
described as critical data fluencies rather than critical data
literacies". Which is the focus?
- The discussion towards the end of the paper made me wonder whether the
users discuss ethical issues around data use and Big Data in terms of any
other platforms at any point. Did this experience with Scratch cause them
to reflect on these issues on other platforms they use (e.g., Instagram)?
If there is any data on this, then the authors may want to discuss this
in a revision.
- The "data comes with assumptions and hidden decisions" section could
have been written more clearly. It was initially unclear how the project
described in this section worked and thus the reactions to this project
did not make complete sense to me, nor was it clear what the assumptions
or hidden decisions were. It was only when the authors later explained
that the algorithm that calculated the number of views was not visible to
the users that I understood that "the "assumptions and hidden decisions"
was not referring to those of the creator of the project, but rather,
those of the developers of these features of Scratch.
- I felt that the quote from Burner (pg. 10) was a bit long and could
have been integrated into the paper better.
- There were a few small grammar mistakes:
page 4: "during the days weeks"
page 7: "more subtler"
page 9: "quantifiable test scores gets"
page 10: "Our findings presents"
Summary:
Significance of the paper's contribution to HCI and the benefit that
others can gain from the contribution: this research suggests unique
pedagogical methods for teaching data science and promoting literacy. It
also provides a deeper understanding of how users experience Big Data,
which is sometimes lacking in discussions about the ethics of Big Data
given the difficulty of researching this topic empirically. Lastly, it
explores the experiences of users who are both producers and analysts of
data--this seems like a relatively unique position for users to be in,
and I think creating more systems like this in the future may reveal new
ways of distributing power among actors in systems that use Big Data.
Originality of the work: this work describes youth views on Big Data and
how they engage with data science. As far as I am aware, there is little
or no research on this topic.
Validity of the work presented: I have confidence in the validity of
these findings, as the authors used many multiple methods to triangulate
their findings.
Presentation clarity: it was very well written and organized in a logical
way that was easy to follow.
Relevant previous work: I cannot comment on whether the authors
referenced all of the relevant literature in education or youth and
social media, but as for literature on Big Data, this seemed complete.
Rebuttal response
I read the rebuttal and felt it addressed my concerns. My main concern
was about inconsistency and ambiguity around the term "critical data
fluencies". The authors suggested simplifying the paper's framing, and I
believe that would make the importance of critical data fluencies clearer
and streamline the paper. If the paper has a simplified framing, then it
may also be possible to include more quotes as the other reviewers
suggested.
I did not change my score.
------------------------ Submission 2826, Review 3 ------------------------
Reviewer: external
Overall rating: 3.5 (scale is 0.5..5; 5 is best)
Expertise
2 (Passing Knowledge)
Recommendation
. . . Between neutral and possibly accept; 3.5
Award Nomination
Your Assessment of this Paper's Contribution to HCI
This works contributes to an understanding of how youth discuss and view
public data analytics of their social media interactions through the lens
of their use of Scratch Community Blocks. This work compares themes that
arise within its findings with existing discourses on Big Data. Finally,
the paper provides recommendations and implications for educators and
designers of learning environments.
The Review
Understanding children’s perspectives- their concerns and their ideas-
about how their personal social-media data is used in public,
particularly as it relates to Big Data uses, is a critical perspective
and a valuable contribution to HCI. The foundations in and discussions on
designs for educational technologies will be of value to HCI community
members who focus on developing learning technologies.
Much of the writing in the work is strong. I particularly appreciate how
clear the authors were in their description of their role in the
research. However, there are structural issues that confuse the document.
The narrative of the paper is disjoint, putting a burden on the reader to
connect the ideas that are presented in the work. The introduction frames
the work in terms of Big Data: how it is used to analyze youth
interactions without allowing youths a voice and opinion on the process.
The framing presented around “highlighting youth’s voices in the
broader conversations of Big Data” is focused on a question the
Introduction and Abstract pose: What should young people know about the
data being collect about them and about the attempts to analyze and
understand these data in ways that can shape their experience? Yet it is
in the background that we discover that the paper will contribute to the
scholarly dialog on what data science education for youth may look like.
The discussion is, in fact, largely focused on recommendations and
implications for educators and designers of learning environments. While
certainly valuable to HCI practitioners who focus on educational
technologies, making it clear how the themes about big data in the
findings and participant perspectives on these themes influenced the
recommendations that were put forth would strengthen and unify the work.
Similarly, returning to the original position of the work, what *should*
young people know about the data being collected about them?
Regarding another structural decision: While choosing to relate each of
the 5 themes in the findings to related work within their descriptions is
somewhat unorthodox within CHI papers, it largely worked. A minor point
is that the paper could do more to set reader’s expectations on this.
(For instance, it was only on my second read of the paper that I realized
the Background section’s introduction was supposed to initially set
this expectation— that text could be more explicit.) A more substantial
point is that the positioning of this work to similar veins of research
(rather than the individual findings) could be improved.
Rebuttal response
I appreciate the author's rebuttal and the clarifications that they
provided. I think that applying a new framing will strengthen the work.
While space may be tight, I do strongly encourage the authors to include
a few more illustrative quotes into their paper (which may be more
helpful to the reader than listing out codes; the proposed clarifications
of the method seems like it would be more than adequate in this regard).
I have raised my score from a 3.0 to a 3.5.

390
chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017.txt Normal file
View File

@ -0,0 +1,390 @@
CHI 2017 Papers and Notes
Reviews of submission #2826: "“Creepy and Very Cool”: Youth
Perspectives on Social Media Data Analytics"
------------------------ Submission 2826, Review 4 ------------------------
Reviewer: primary (1AC)
Expertise
3 (Knowledgeable)
Recommendation
. . . Between neutral and possibly accept; 3.5
Award Nomination
If accepted, this paper would not be among the top 20% of papers presented at CHI
Your Assessment of this Paper's Contribution to HCI
This paper reports a qualitative study of youth interaction with social
media usage data. In particular participants used an extension of the
Scratch programming environment, which allows them to write programs to
process data about other participants’ behavior on such extended
Scratch system. This is somewhat similar to the kind of data available on
other social networks.
The data, analyzed through grounded theory, illustrates how youths make
sense of the data and its implications on privacy.
The Review
1AC: The Meta-Review
All reviewers are generally positive about the contribution made by this
paper, as well its timeliness and novelty. R2 is particularly
enthusiastic about the paper, while the other two reviewers flag some
concerns. While all reviewers agree that the paper is generally well
written, they all suggest some re-framing, with R1 and R3 being
particularly critical about this matter.
In particular, all reviewers suggest that the paper would benefit from
reducing the emphasis on big data, and instead narrowing the focus of the
introduction. R1 offers a very practical suggestion, to focus on
“learning about one’s data is the most interesting”. This is the
key issue that the authors need to address in their rebuttal.
Other issues that would be good to see addressed in the rebuttal include:
- clarification of the analysis; more information should be provided
about the codes used for different parts of the data (e.g. interviews vs.
survey responses), and how these were combined (R1); moreover, it would
be good to see a justification of the use of “deductive codes drawn
from existing theory” within a grounded theory approach (which is
normally focused just on inductive codes emerging from the data) (AC1).
- The introduction should more clearly outline the rest of the paper, to
prepare readers for what to expect (R3)
- discussion of how general or specific the reported findings are (R1)
- more information about whether participants translated some of their
insight from Scratch to other social media platforms (R1, R2)
Rebuttal response
------------------------ Submission 2826, Review 5 ------------------------
Reviewer: secondary (2AC)
Expertise
3 (Knowledgeable)
Recommendation
. . . Between neutral and possibly accept; 3.5
Award Nomination
Your Assessment of this Paper's Contribution to HCI
The Review
This paper addresses a timely and important issue. The reviewers are in
agreement that there is much value in this work. In order to move
forward, I would recommend that the authors focus on two main issues
present in the reviewers' comments in their rebuttal.
First, all reviewers comment on the confusion regarding the framing of
the work, which has the outcome of obscuring the main contributions of
the research. The intended focus should be clarified.
Second, the methods used should be further detailed. Particularly,
answering the questions asked by R1 would go a long way towards
convincing readers that the work done in this project is reliable and
valid.
Rebuttal response
------------------------ Submission 2826, Review 1 ------------------------
Reviewer: external
Expertise
4 (Expert )
Recommendation
. . . Between neutral and possibly accept; 3.5
Award Nomination
Your Assessment of this Paper's Contribution to HCI
This paper contributes into how young people think about and reflect on
their own data in the context of Scratch. HCI researchers could use this
to better design technologies for young people, and to think about
privacy and related theoretical issues related to youth and data.
The Review
The premise of this paper is promising. I agree youth perspectives on big
data are understudied and I’m very excited to see efforts to explore
their perspectives. I find the choice of using Scratch to study youths’
perspectives an intriguing one. The main question I have is that Scratch
is an educational platform, hosted by MIT, and it does not advertise or
sell data to anyone (as far as I know). This puts it in a dramatically
different category of platforms compared to stereotypical big data sites
like Facebook or Twitter or Google or even educational sites like MOOCs
or Canvas, which are ultimately for profit industries. How would the
authors frame the contributions here given that big data concerns have
largely been about these other categories? (My personal opinion is that
the big data framing actually does a disservice to the narrative by
distracting the reader. This is about how youth learn about and make
sense of data that is collected and displayed about themselves).
The authors collected so much data, I would have liked to see more of it
to show evidence of their claims. For example, the statement “we
observed how Scratch users are grasping the implications of what it means
to have this data out there” – I wanted to know what are they
grasping at? How are they grasping? We are given one “creepy but
cool” quote which is also the title of the paper, and it feels
cherry-picked without more data to support the claim.
I would have liked to see a more systematic and in-depth analysis of the
survey data. 400 responses is a lot and I think it’s worth developing a
codebook and coding the data for key themes then telling the reader about
those. As it is now, the concern about cherry picking data remains
salient because the description of the data coding is hard to discern.
Did the author code all of the data they collected? (interviews, survey,
log files, workshop notes) It seems like it would be very hard to develop
a single codebook for all of these different data sources. And either
way, can the authors tell us about what kinds of codes were in the
codebook, how often key codes showed up, etc?
How representative do the authors think the data they present in this
paper is of youths’ attitudes towards big data generally? There’s
some concern about biases here. The authors isolated the most active
Scratch users, then sample from the more engaged among them (presumably,
those who agree to do a survey are particularly active Scratch users).
What would the authors say we learned that can be generalized versus
cannot?
To return to the “big data” framing critique, it seems like the
authors themselves aren’t exactly sure what it is they’re trying to
articulate sometimes. They refer to data literacy, social media data
analytics, data science. These terms are all kind of loaded and they’re
used in various ways throughout the paper. I would encourage the authors
to pick a clearer narrative and terminology (I think the learning about
one’s data is the most interesting—“data science”, “big
data”, and “social media” aren’t really great fits for this
study).
In general, I’m excited by the ideas and motivation of this work. I
would be more positive if the data analysis process were more transparent
and clear throughout, and if the authors could make a more focused
contribution without relying on (what I would argue are) overused popular
terms.
Rebuttal response
------------------------ Submission 2826, Review 2 ------------------------
Reviewer: external
Expertise
3 (Knowledgeable)
Recommendation
Strong Accept: I would argue strongly for accepting this paper; 5.0
Award Nomination
If accepted, this paper would be among the top 5% of papers presented at CHI (Best Paper nomination)
Your Assessment of this Paper's Contribution to HCI
This paper looks at youth who used Scratch "blocks" to access metadata on
other users. The programs they created led the users to think deeply
about issues with Big Data. This study provides an understanding of 1)
youth experiences of Big Data (which has largely been understudied) 2) an
exploration of a system where users both produce and analyze metadata 3)
pedagogical suggestions for teaching data science.
The Review
What I liked:
- There was a lot to like about this paper! It was surprising to me that
the issues with Big Data that scholars have noted were so readily
identified by youth on Scratch Community Blocks. The insights about how
children and teens view issues of Big Data were in line with things that
researchers have noted in their own critiques, but some of these issues
were presented in a new light when viewed through youths' perspectives.
- I was initially skeptical about how such simple changes to Scratch
could provoke so much reflection from users, but I was thoroughly
convinced by the end of the paper that even relatively small changes to
Scratch had a significant impact on how users understood their
relationship to their data and the ethics of algorithms.
- I was impressed with the rather comprehensive methods used by the
researchers.
- The authors made strong arguments for using these sorts of techniques
for teaching about the ethics of data science.
Issues:
- I was not clear on the importance of discussing "critical data fluency"
because it was not returned to explicitly later in the paper. The authors
seemed to position this concept as a contribution to discussions about
literacy, so I expected to see more about it.
- The authors say "we focus on what can be termed as critical data
literacies" and then say that "the themes that we find are better
described as critical data fluencies rather than critical data
literacies". Which is the focus?
- The discussion towards the end of the paper made me wonder whether the
users discuss ethical issues around data use and Big Data in terms of any
other platforms at any point. Did this experience with Scratch cause them
to reflect on these issues on other platforms they use (e.g., Instagram)?
If there is any data on this, then the authors may want to discuss this
in a revision.
- The "data comes with assumptions and hidden decisions" section could
have been written more clearly. It was initially unclear how the project
described in this section worked and thus the reactions to this project
did not make complete sense to me, nor was it clear what the assumptions
or hidden decisions were. It was only when the authors later explained
that the algorithm that calculated the number of views was not visible to
the users that I understood that "the "assumptions and hidden decisions"
was not referring to those of the creator of the project, but rather,
those of the developers of these features of Scratch.
- I felt that the quote from Burner (pg. 10) was a bit long and could
have been integrated into the paper better.
- There were a few small grammar mistakes:
page 4: "during the days weeks"
page 7: "more subtler"
page 9: "quantifiable test scores gets"
page 10: "Our findings presents"
Summary:
Significance of the paper's contribution to HCI and the benefit that
others can gain from the contribution: this research suggests unique
pedagogical methods for teaching data science and promoting literacy. It
also provides a deeper understanding of how users experience Big Data,
which is sometimes lacking in discussions about the ethics of Big Data
given the difficulty of researching this topic empirically. Lastly, it
explores the experiences of users who are both producers and analysts of
data--this seems like a relatively unique position for users to be in,
and I think creating more systems like this in the future may reveal new
ways of distributing power among actors in systems that use Big Data.
Originality of the work: this work describes youth views on Big Data and
how they engage with data science. As far as I am aware, there is little
or no research on this topic.
Validity of the work presented: I have confidence in the validity of
these findings, as the authors used many multiple methods to triangulate
their findings.
Presentation clarity: it was very well written and organized in a logical
way that was easy to follow.
Relevant previous work: I cannot comment on whether the authors
referenced all of the relevant literature in education or youth and
social media, but as for literature on Big Data, this seemed complete.
Rebuttal response
------------------------ Submission 2826, Review 3 ------------------------
Reviewer: external
Expertise
2 (Passing Knowledge)
Recommendation
Neutral: I am unable to argue for accepting or rejecting this paper; 3.0
Award Nomination
Your Assessment of this Paper's Contribution to HCI
This works contributes to an understanding of how youth discuss and view
public data analytics of their social media interactions through the lens
of their use of Scratch Community Blocks. This work compares themes that
arise within its findings with existing discourses on Big Data. Finally,
the paper provides recommendations and implications for educators and
designers of learning environments.
The Review
Understanding children’s perspectives- their concerns and their ideas-
about how their personal social-media data is used in public,
particularly as it relates to Big Data uses, is a critical perspective
and a valuable contribution to HCI. The foundations in and discussions on
designs for educational technologies will be of value to HCI community
members who focus on developing learning technologies.
Much of the writing in the work is strong. I particularly appreciate how
clear the authors were in their description of their role in the
research. However, there are structural issues that confuse the document.
The narrative of the paper is disjoint, putting a burden on the reader to
connect the ideas that are presented in the work. The introduction frames
the work in terms of Big Data: how it is used to analyze youth
interactions without allowing youths a voice and opinion on the process.
The framing presented around “highlighting youth’s voices in the
broader conversations of Big Data” is focused on a question the
Introduction and Abstract pose: What should young people know about the
data being collect about them and about the attempts to analyze and
understand these data in ways that can shape their experience? Yet it is
in the background that we discover that the paper will contribute to the
scholarly dialog on what data science education for youth may look like.
The discussion is, in fact, largely focused on recommendations and
implications for educators and designers of learning environments. While
certainly valuable to HCI practitioners who focus on educational
technologies, making it clear how the themes about big data in the
findings and participant perspectives on these themes influenced the
recommendations that were put forth would strengthen and unify the work.
Similarly, returning to the original position of the work, what *should*
young people know about the data being collected about them?
Regarding another structural decision: While choosing to relate each of
the 5 themes in the findings to related work within their descriptions is
somewhat unorthodox within CHI papers, it largely worked. A minor point
is that the paper could do more to set reader’s expectations on this.
(For instance, it was only on my second read of the paper that I realized
the Background section’s introduction was supposed to initially set
this expectation— that text could be more explicit.) A more substantial
point is that the positioning of this work to similar veins of research
(rather than the individual findings) could be improved.
Rebuttal response

25
chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-revision_summary.txt Normal file
View File

@ -0,0 +1,25 @@
* We have changed the title of the paper to "Youth Perspectives on Critical Data Literacies" (as per R1's response to our rebuttal and as discussed in our letter to 1AC)
* We have e the introduction and framing to focus on critical data literacies rather than social media, data science etc. As a result of this change, we have standardized on the term literacies throughout. The term "fluency" is no longer even mentioned in the manuscript.
* We have added and expanded sections about our grounded theory methodology. We have reworked much of our "Data and Methodology" section to add text to make our methodology more clear. In particular, we have made it clear that we closely followed Charmaz's influential textbook on grounded theory which is different, in some respects, from Corbin and Strauss' seminal text. We have also made it clear that we used a single codebook and that we coded several "earlier" sources of data more openly than data (like the survey) that came in later.
* In particular, we have expanded our discussion on our use of deductive codes and the process by which we derived them. As we suggested in our rebuttal, we used both "sensitizing" codes drawn from theory during initial coding as well as later-stage iterative recoding of data using codes informed by theory relevant to emergent themes.
* Also, to better illustrate of the process by which we arrived at the themes we describe in our work, we added a paragraph to our data and methodology section that provides examples of codes we derived from open coding and themes which emerged from the codes. We hope that this makes our process and methodology much more transparent.
* We added a clear outline of the rest of the paper to the end of our introduction..
* We have tried to set expectations about the structure of the findings subsections more explicitly. As per R3, we have added a sentence to the paragraph laying out our plan for the paper to explain to readers that we structure our findings so that we relate each one to prior work.
* We have edited our discussion of generalizability in our introduction to make the limitations, in this regard, more clear. We have also added a brief discussion of how we do not know how kids might translate their insights into Scratch to other social media platforms.
* We have a number of new new quotes to the paper's findings section and concluded the section on each literacy with a paragraph that explicitly ties it to existing theory and reiterates the connection to critical data literacies.
* We have clarified the data comes with assumptions and hidden decisions sections as requested
* We have shortened and integrated the quote from Bruner in our discussion.
* We have shortened our discussion and conclusion and merged them into a single section.
* We have fixed all the grammar mistakes that were pointed out by the reviewers.