diff --git a/chi_rebuttals/2017-critical_data_literacies/README.txt b/chi_rebuttals/2017-critical_data_literacies/README.txt new file mode 100644 index 0000000..2dd60d6 --- /dev/null +++ b/chi_rebuttals/2017-critical_data_literacies/README.txt @@ -0,0 +1,7 @@ +Rebuttal for: + +Hautea, Samantha, Sayamindu Dasgupta, and Benjamin Mako +Hill. 2017. “Youth Perspectives on Critical Data Literacies.” In +Proceedings of the 2017 CHI Conference on Human Factors in Computing +Systems (CHI ’17), 919–930. New York, New York: +ACM. https://doi.org/10.1145/3025453.3025823. diff --git a/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-rebuttal-CHI2017.txt b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-rebuttal-CHI2017.txt new file mode 100644 index 0000000..ef3a1f3 --- /dev/null +++ b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-rebuttal-CHI2017.txt @@ -0,0 +1,34 @@ +We thank the reviewers for their careful attention and feedback. Below, we describe adjustments we plan to make that we believe will address the reviewers central concerns. Although relatively minor, we agree that these changes will improve the manuscript enormously. We are confident that we can tighten our text and remove discussion of less relevant concepts (as described below) to accommodate the proposed additions. + +1. +R1 and R4 suggest that our work is unfocused and discusses too many loaded terms: data literacies, data fluencies, social media, big data, data analytics, and data science. As per R1, we will revise our paper so that its contribution is more narrowly framed in terms of “critical data fluencies.” As per R2 and R3, we will return to this concept in the discussion and refer to it when presenting our findings to offer a more cohesive narrative. + +As per R1 and R4, we will remove discussion of social media and data science. As per R1, we will de-emphasize discussion of big data except where studies of big data complement our findings in ways that were cited by R2 as central to our work’s contribution (e.g. connections to boyd and Crawford). + +Given these changes, we feel that our previous title is inappropriate. We will retitle the manuscript “Youth Perspectives of Critical Data Fluencies.” + +2. +A concern given this reframing is R2’s point that we were inconsistent in our use of the terms literacies and fluencies. We will standardize on the term fluencies and add citations to work that advocates for this term over literacies (e.g. Resnick, Rusk, & Cooke, 1998). + +3. +As per R1 and R4, we will clarify our methodology. We will explain that our approach was drawn directly from Charmaz (2006)’s textbook on grounded theory (GT). As per Charmaz, we iteratively produced a single codebook with various sources of data as a means of triangulation. We will explain that although we coded all data, we coded “earlier” data more openly and heavily and coded later data (including the survey) more lightly and with more reliance on existing codes. Our QDA software (Dedoose) allowed us to code interview transcripts, images, and field notes in the same interface. + +R1 and R4 asked for details about our codes. Although we have not seen it done, we will export the full list of codes we have used and publish it as supplementary material if requested. + +4. +As per R4, we will explain how we used deductive codes. Although some early descriptions of GT advocate coding with no preconceived concepts, we followed Charmaz’s call for “an open mind, not an empty head” (pg. 174) and the use of both “sensitizing” codes drawn from theory during initial coding as well as later-stage iterative recoding of data using codes informed by theory relevant to emergent themes. We will make this process explicit in our methodology section. + +5. +R3 and R4 suggested that our introduction should clearly outline the rest of our paper. We will edit our introduction to do this. + +6. +R1 and R4 asked us to reflect on how our findings might generalize and how broadly representative we believe the concerns raised by kids in our sample are. We will explain that although we believe that our themes are a valid description of the major ways that kids discussed the social implications of data in our sample, they reflect the aggregate observations of active and engaged Scratch users. We will explain that no child raised every issue and that it is possible that a system with different affordances might surface different concerns. Moreover, we will remind readers that open questions around generalizability are a frequently-cited limitation of GT. + +7. +In a related sense, R1, R2, and R4 asked us whether participants translated their insights from Scratch to other social media platforms. We will add text to our discussion to explain that, although there are similarities between Scratch and other systems that might support this, we have little evidence that this occurred. We considered asking questions about this on our survey but did not because we felt that doing so would involve collecting evidence that users under 13 had broken social media websites’ ToS which usually disallows these users. We will raise this as a limitation of our study. + +8. +R1 requested that we include more quotes to address concerns about cherry-picking. As per Charmaz, we selected quotes that we felt were clear reflections of the themes we identified. Where possible — including the “creepy but cool” example — we will include additional quotes. That said, we are limited by space and even with improvement in this regard, our paper will only showcase a tiny amount of our data. We hope that several additional quotes and more detailed description of our methodology can alleviate R1’s concerns. + +9. +We will fix all the typos and grammatical errors pointed out by the reviewers and have our manuscript professionally proof-read. diff --git a/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017-postpc.txt b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017-postpc.txt new file mode 100644 index 0000000..44c85ab --- /dev/null +++ b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017-postpc.txt @@ -0,0 +1,444 @@ +CHI 2017 Papers and Notes + +Reviews of submission #2826: "“Creepy and Very Cool”: Youth +Perspectives on Social Media Data Analytics" + +------------------------ Submission 2826, Review 4 ------------------------ + +Reviewer: primary (1AC) +Overall rating: 4 (scale is 0.5..5; 5 is best) + +Expertise + + 2 (Passing Knowledge) + +Recommendation + + Possibly Accept: I would argue for accepting this paper; 4.0 + +Award Nomination + + If accepted, this paper would not be among the top 20% of papers presented at CHI + +Your Assessment of this Paper's Contribution to HCI + + This paper reports a qualitative study of youth interaction with social + media usage data. In particular participants used an extension of the + Scratch programming environment, which allows them to write programs to + process data about other participants’ behavior on such extended + Scratch system. This is somewhat similar to the kind of data available on + other social networks. + The data, analyzed through grounded theory, illustrates how youths make + sense of the data and its implications on privacy. + + +The Review + + +1AC: The Meta-Review + + All reviewers are generally positive about the contribution made by this + paper, as well its timeliness and novelty. R2 is particularly + enthusiastic about the paper, while the other two reviewers flag some + concerns. While all reviewers agree that the paper is generally well + written, they all suggest some re-framing, with R1 and R3 being + particularly critical about this matter. + + In particular, all reviewers suggest that the paper would benefit from + reducing the emphasis on big data, and instead narrowing the focus of the + introduction. R1 offers a very practical suggestion, to focus on + “learning about one’s data is the most interesting”. This is the + key issue that the authors need to address in their rebuttal. + + Other issues that would be good to see addressed in the rebuttal include: + - clarification of the analysis; more information should be provided + about the codes used for different parts of the data (e.g. interviews vs. + survey responses), and how these were combined (R1); moreover, it would + be good to see a justification of the use of “deductive codes drawn + from existing theory” within a grounded theory approach (which is + normally focused just on inductive codes emerging from the data) (AC1). + - The introduction should more clearly outline the rest of the paper, to + prepare readers for what to expect (R3) + - discussion of how general or specific the reported findings are (R1) + - more information about whether participants translated some of their + insight from Scratch to other social media platforms (R1, R2) + + + *** Comments after rebuttal and PC meeting *** + + Congratulations to the authors on their paper’s conditional acceptance. + To gain final approval for acceptance, the authors should address the + reviewer comments and implement the changes as proposed in the rebuttal. + The deadline for PCS submission for consideration for final acceptance is + 6 January 2017 (20:00 EST). + +Rebuttal response + + The rebuttal convinced me that the authors can address the issues raised + by reviewers and improve the paper to a level suitable for publication at + CHI. In light of the rebuttal, I believe the paper can make a timely and + important contribution at CHI, so I raise my score from 3.5 to 4. + + The external reviewers also received positively the rebuttal, and two of + them raised their scores. + + +------------------------ Submission 2826, Review 5 ------------------------ + +Reviewer: secondary (2AC) +Overall rating: 4 (scale is 0.5..5; 5 is best) + +Expertise + + 3 (Knowledgeable) + +Recommendation + + Possibly Accept: I would argue for accepting this paper; 4.0 + +Award Nomination + + + +Your Assessment of this Paper's Contribution to HCI + + +The Review + + This paper addresses a timely and important issue. The reviewers are in + agreement that there is much value in this work. In order to move + forward, I would recommend that the authors focus on two main issues + present in the reviewers' comments in their rebuttal. + + First, all reviewers comment on the confusion regarding the framing of + the work, which has the outcome of obscuring the main contributions of + the research. The intended focus should be clarified. + + Second, the methods used should be further detailed. Particularly, + answering the questions asked by R1 would go a long way towards + convincing readers that the work done in this project is reliable and + valid. + +1AC: The Meta-Review + + +Rebuttal response + + + +------------------------ Submission 2826, Review 1 ------------------------ + +Reviewer: external +Overall rating: 4 (scale is 0.5..5; 5 is best) + +Expertise + + 4 (Expert ) + +Recommendation + + Possibly Accept: I would argue for accepting this paper; 4.0 + +Award Nomination + + + +Your Assessment of this Paper's Contribution to HCI + + This paper contributes into how young people think about and reflect on + their own data in the context of Scratch. HCI researchers could use this + to better design technologies for young people, and to think about + privacy and related theoretical issues related to youth and data. + +The Review + + The premise of this paper is promising. I agree youth perspectives on big + data are understudied and I’m very excited to see efforts to explore + their perspectives. I find the choice of using Scratch to study youths’ + perspectives an intriguing one. The main question I have is that Scratch + is an educational platform, hosted by MIT, and it does not advertise or + sell data to anyone (as far as I know). This puts it in a dramatically + different category of platforms compared to stereotypical big data sites + like Facebook or Twitter or Google or even educational sites like MOOCs + or Canvas, which are ultimately for profit industries. How would the + authors frame the contributions here given that big data concerns have + largely been about these other categories? (My personal opinion is that + the big data framing actually does a disservice to the narrative by + distracting the reader. This is about how youth learn about and make + sense of data that is collected and displayed about themselves). + + The authors collected so much data, I would have liked to see more of it + to show evidence of their claims. For example, the statement “we + observed how Scratch users are grasping the implications of what it means + to have this data out there” – I wanted to know what are they + grasping at? How are they grasping? We are given one “creepy but + cool” quote which is also the title of the paper, and it feels + cherry-picked without more data to support the claim. + + I would have liked to see a more systematic and in-depth analysis of the + survey data. 400 responses is a lot and I think it’s worth developing a + codebook and coding the data for key themes then telling the reader about + those. As it is now, the concern about cherry picking data remains + salient because the description of the data coding is hard to discern. + Did the author code all of the data they collected? (interviews, survey, + log files, workshop notes) It seems like it would be very hard to develop + a single codebook for all of these different data sources. And either + way, can the authors tell us about what kinds of codes were in the + codebook, how often key codes showed up, etc? + + How representative do the authors think the data they present in this + paper is of youths’ attitudes towards big data generally? There’s + some concern about biases here. The authors isolated the most active + Scratch users, then sample from the more engaged among them (presumably, + those who agree to do a survey are particularly active Scratch users). + What would the authors say we learned that can be generalized versus + cannot? + + To return to the “big data” framing critique, it seems like the + authors themselves aren’t exactly sure what it is they’re trying to + articulate sometimes. They refer to data literacy, social media data + analytics, data science. These terms are all kind of loaded and they’re + used in various ways throughout the paper. I would encourage the authors + to pick a clearer narrative and terminology (I think the learning about + one’s data is the most interesting—“data science”, “big + data”, and “social media” aren’t really great fits for this + study). + + In general, I’m excited by the ideas and motivation of this work. I + would be more positive if the data analysis process were more transparent + and clear throughout, and if the authors could make a more focused + contribution without relying on (what I would argue are) overused popular + terms. + +Rebuttal response + + I have read the rebuttal. I appreciate the authors' explanation of their + data analysis and would expect to see that and hopefully more details in + the published paper. We don't need to see all of your codes, but we do + need enough to understand what you did and that there was a rigorous and + systematic process followed. I think whether you add some example codes + to the extended methods detail or add them in an appendix is up to you. + + I personally don't like the "critical data fluencies" framing as the + predominant framing. It's hard to parse and may not be meaningful to a + broad audience of readers (I don't find references to it in a quick + Google Scholar search). I realize previously I said the framings were too + broad and now I'm saying it's not broad enough. Nonetheless, I would + encourage the authors to take some time to rethink what they want their + overarching contribution and framing to be. The major point in our + reviews which the authors grokked is to figure out what that story is and + make it clear throughout. I've changed my review from a 3.5 to a 4. + + +------------------------ Submission 2826, Review 2 ------------------------ + +Reviewer: external +Overall rating: 5 (scale is 0.5..5; 5 is best) + +Expertise + + 3 (Knowledgeable) + +Recommendation + + Strong Accept: I would argue strongly for accepting this paper; 5.0 + +Award Nomination + + If accepted, this paper would be among the top 5% of papers presented at CHI (Best Paper nomination) + +Your Assessment of this Paper's Contribution to HCI + + This paper looks at youth who used Scratch "blocks" to access metadata on + other users. The programs they created led the users to think deeply + about issues with Big Data. This study provides an understanding of 1) + youth experiences of Big Data (which has largely been understudied) 2) an + exploration of a system where users both produce and analyze metadata 3) + pedagogical suggestions for teaching data science. + +The Review + + What I liked: + + - There was a lot to like about this paper! It was surprising to me that + the issues with Big Data that scholars have noted were so readily + identified by youth on Scratch Community Blocks. The insights about how + children and teens view issues of Big Data were in line with things that + researchers have noted in their own critiques, but some of these issues + were presented in a new light when viewed through youths' perspectives. + + - I was initially skeptical about how such simple changes to Scratch + could provoke so much reflection from users, but I was thoroughly + convinced by the end of the paper that even relatively small changes to + Scratch had a significant impact on how users understood their + relationship to their data and the ethics of algorithms. + + - I was impressed with the rather comprehensive methods used by the + researchers. + + - The authors made strong arguments for using these sorts of techniques + for teaching about the ethics of data science. + + + Issues: + + - I was not clear on the importance of discussing "critical data fluency" + because it was not returned to explicitly later in the paper. The authors + seemed to position this concept as a contribution to discussions about + literacy, so I expected to see more about it. + + - The authors say "we focus on what can be termed as critical data + literacies" and then say that "the themes that we find are better + described as critical data fluencies rather than critical data + literacies". Which is the focus? + + - The discussion towards the end of the paper made me wonder whether the + users discuss ethical issues around data use and Big Data in terms of any + other platforms at any point. Did this experience with Scratch cause them + to reflect on these issues on other platforms they use (e.g., Instagram)? + If there is any data on this, then the authors may want to discuss this + in a revision. + + - The "data comes with assumptions and hidden decisions" section could + have been written more clearly. It was initially unclear how the project + described in this section worked and thus the reactions to this project + did not make complete sense to me, nor was it clear what the assumptions + or hidden decisions were. It was only when the authors later explained + that the algorithm that calculated the number of views was not visible to + the users that I understood that "the "assumptions and hidden decisions" + was not referring to those of the creator of the project, but rather, + those of the developers of these features of Scratch. + + - I felt that the quote from Burner (pg. 10) was a bit long and could + have been integrated into the paper better. + + - There were a few small grammar mistakes: + page 4: "during the days weeks" + page 7: "more subtler" + page 9: "quantifiable test scores gets" + page 10: "Our findings presents" + + + Summary: + + Significance of the paper's contribution to HCI and the benefit that + others can gain from the contribution: this research suggests unique + pedagogical methods for teaching data science and promoting literacy. It + also provides a deeper understanding of how users experience Big Data, + which is sometimes lacking in discussions about the ethics of Big Data + given the difficulty of researching this topic empirically. Lastly, it + explores the experiences of users who are both producers and analysts of + data--this seems like a relatively unique position for users to be in, + and I think creating more systems like this in the future may reveal new + ways of distributing power among actors in systems that use Big Data. + + Originality of the work: this work describes youth views on Big Data and + how they engage with data science. As far as I am aware, there is little + or no research on this topic. + + Validity of the work presented: I have confidence in the validity of + these findings, as the authors used many multiple methods to triangulate + their findings. + + Presentation clarity: it was very well written and organized in a logical + way that was easy to follow. + + Relevant previous work: I cannot comment on whether the authors + referenced all of the relevant literature in education or youth and + social media, but as for literature on Big Data, this seemed complete. + +Rebuttal response + + I read the rebuttal and felt it addressed my concerns. My main concern + was about inconsistency and ambiguity around the term "critical data + fluencies". The authors suggested simplifying the paper's framing, and I + believe that would make the importance of critical data fluencies clearer + and streamline the paper. If the paper has a simplified framing, then it + may also be possible to include more quotes as the other reviewers + suggested. + + I did not change my score. + + +------------------------ Submission 2826, Review 3 ------------------------ + +Reviewer: external +Overall rating: 3.5 (scale is 0.5..5; 5 is best) + +Expertise + + 2 (Passing Knowledge) + +Recommendation + + . . . Between neutral and possibly accept; 3.5 + +Award Nomination + + + +Your Assessment of this Paper's Contribution to HCI + + This works contributes to an understanding of how youth discuss and view + public data analytics of their social media interactions through the lens + of their use of Scratch Community Blocks. This work compares themes that + arise within its findings with existing discourses on Big Data. Finally, + the paper provides recommendations and implications for educators and + designers of learning environments. + +The Review + + Understanding children’s perspectives- their concerns and their ideas- + about how their personal social-media data is used in public, + particularly as it relates to Big Data uses, is a critical perspective + and a valuable contribution to HCI. The foundations in and discussions on + designs for educational technologies will be of value to HCI community + members who focus on developing learning technologies. + + Much of the writing in the work is strong. I particularly appreciate how + clear the authors were in their description of their role in the + research. However, there are structural issues that confuse the document. + + + The narrative of the paper is disjoint, putting a burden on the reader to + connect the ideas that are presented in the work. The introduction frames + the work in terms of Big Data: how it is used to analyze youth + interactions without allowing youths a voice and opinion on the process. + The framing presented around “highlighting youth’s voices in the + broader conversations of Big Data” is focused on a question the + Introduction and Abstract pose: What should young people know about the + data being collect about them and about the attempts to analyze and + understand these data in ways that can shape their experience? Yet it is + in the background that we discover that the paper will contribute to the + scholarly dialog on what data science education for youth may look like. + The discussion is, in fact, largely focused on recommendations and + implications for educators and designers of learning environments. While + certainly valuable to HCI practitioners who focus on educational + technologies, making it clear how the themes about big data in the + findings and participant perspectives on these themes influenced the + recommendations that were put forth would strengthen and unify the work. + Similarly, returning to the original position of the work, what *should* + young people know about the data being collected about them? + + Regarding another structural decision: While choosing to relate each of + the 5 themes in the findings to related work within their descriptions is + somewhat unorthodox within CHI papers, it largely worked. A minor point + is that the paper could do more to set reader’s expectations on this. + (For instance, it was only on my second read of the paper that I realized + the Background section’s introduction was supposed to initially set + this expectation— that text could be more explicit.) A more substantial + point is that the positioning of this work to similar veins of research + (rather than the individual findings) could be improved. + +Rebuttal response + + I appreciate the author's rebuttal and the clarifications that they + provided. I think that applying a new framing will strengthen the work. + While space may be tight, I do strongly encourage the authors to include + a few more illustrative quotes into their paper (which may be more + helpful to the reader than listing out codes; the proposed clarifications + of the method seems like it would be more than adequate in this regard). + I have raised my score from a 3.0 to a 3.5. + + + diff --git a/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017.txt b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017.txt new file mode 100644 index 0000000..fa7fd01 --- /dev/null +++ b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-reviews-CHI2017.txt @@ -0,0 +1,390 @@ +CHI 2017 Papers and Notes + +Reviews of submission #2826: "“Creepy and Very Cool”: Youth +Perspectives on Social Media Data Analytics" + +------------------------ Submission 2826, Review 4 ------------------------ + +Reviewer: primary (1AC) + +Expertise + + 3 (Knowledgeable) + +Recommendation + + . . . Between neutral and possibly accept; 3.5 + +Award Nomination + + If accepted, this paper would not be among the top 20% of papers presented at CHI + +Your Assessment of this Paper's Contribution to HCI + + This paper reports a qualitative study of youth interaction with social + media usage data. In particular participants used an extension of the + Scratch programming environment, which allows them to write programs to + process data about other participants’ behavior on such extended + Scratch system. This is somewhat similar to the kind of data available on + other social networks. + The data, analyzed through grounded theory, illustrates how youths make + sense of the data and its implications on privacy. + + +The Review + + +1AC: The Meta-Review + + All reviewers are generally positive about the contribution made by this + paper, as well its timeliness and novelty. R2 is particularly + enthusiastic about the paper, while the other two reviewers flag some + concerns. While all reviewers agree that the paper is generally well + written, they all suggest some re-framing, with R1 and R3 being + particularly critical about this matter. + + In particular, all reviewers suggest that the paper would benefit from + reducing the emphasis on big data, and instead narrowing the focus of the + introduction. R1 offers a very practical suggestion, to focus on + “learning about one’s data is the most interesting”. This is the + key issue that the authors need to address in their rebuttal. + + Other issues that would be good to see addressed in the rebuttal include: + - clarification of the analysis; more information should be provided + about the codes used for different parts of the data (e.g. interviews vs. + survey responses), and how these were combined (R1); moreover, it would + be good to see a justification of the use of “deductive codes drawn + from existing theory” within a grounded theory approach (which is + normally focused just on inductive codes emerging from the data) (AC1). + - The introduction should more clearly outline the rest of the paper, to + prepare readers for what to expect (R3) + - discussion of how general or specific the reported findings are (R1) + - more information about whether participants translated some of their + insight from Scratch to other social media platforms (R1, R2) + + + +Rebuttal response + + + +------------------------ Submission 2826, Review 5 ------------------------ + +Reviewer: secondary (2AC) + +Expertise + + 3 (Knowledgeable) + +Recommendation + + . . . Between neutral and possibly accept; 3.5 + +Award Nomination + + + +Your Assessment of this Paper's Contribution to HCI + + +The Review + + This paper addresses a timely and important issue. The reviewers are in + agreement that there is much value in this work. In order to move + forward, I would recommend that the authors focus on two main issues + present in the reviewers' comments in their rebuttal. + + First, all reviewers comment on the confusion regarding the framing of + the work, which has the outcome of obscuring the main contributions of + the research. The intended focus should be clarified. + + Second, the methods used should be further detailed. Particularly, + answering the questions asked by R1 would go a long way towards + convincing readers that the work done in this project is reliable and + valid. + +Rebuttal response + + + +------------------------ Submission 2826, Review 1 ------------------------ + +Reviewer: external + +Expertise + + 4 (Expert ) + +Recommendation + + . . . Between neutral and possibly accept; 3.5 + +Award Nomination + + + +Your Assessment of this Paper's Contribution to HCI + + This paper contributes into how young people think about and reflect on + their own data in the context of Scratch. HCI researchers could use this + to better design technologies for young people, and to think about + privacy and related theoretical issues related to youth and data. + +The Review + + The premise of this paper is promising. I agree youth perspectives on big + data are understudied and I’m very excited to see efforts to explore + their perspectives. I find the choice of using Scratch to study youths’ + perspectives an intriguing one. The main question I have is that Scratch + is an educational platform, hosted by MIT, and it does not advertise or + sell data to anyone (as far as I know). This puts it in a dramatically + different category of platforms compared to stereotypical big data sites + like Facebook or Twitter or Google or even educational sites like MOOCs + or Canvas, which are ultimately for profit industries. How would the + authors frame the contributions here given that big data concerns have + largely been about these other categories? (My personal opinion is that + the big data framing actually does a disservice to the narrative by + distracting the reader. This is about how youth learn about and make + sense of data that is collected and displayed about themselves). + + The authors collected so much data, I would have liked to see more of it + to show evidence of their claims. For example, the statement “we + observed how Scratch users are grasping the implications of what it means + to have this data out there” – I wanted to know what are they + grasping at? How are they grasping? We are given one “creepy but + cool” quote which is also the title of the paper, and it feels + cherry-picked without more data to support the claim. + + I would have liked to see a more systematic and in-depth analysis of the + survey data. 400 responses is a lot and I think it’s worth developing a + codebook and coding the data for key themes then telling the reader about + those. As it is now, the concern about cherry picking data remains + salient because the description of the data coding is hard to discern. + Did the author code all of the data they collected? (interviews, survey, + log files, workshop notes) It seems like it would be very hard to develop + a single codebook for all of these different data sources. And either + way, can the authors tell us about what kinds of codes were in the + codebook, how often key codes showed up, etc? + + How representative do the authors think the data they present in this + paper is of youths’ attitudes towards big data generally? There’s + some concern about biases here. The authors isolated the most active + Scratch users, then sample from the more engaged among them (presumably, + those who agree to do a survey are particularly active Scratch users). + What would the authors say we learned that can be generalized versus + cannot? + + To return to the “big data” framing critique, it seems like the + authors themselves aren’t exactly sure what it is they’re trying to + articulate sometimes. They refer to data literacy, social media data + analytics, data science. These terms are all kind of loaded and they’re + used in various ways throughout the paper. I would encourage the authors + to pick a clearer narrative and terminology (I think the learning about + one’s data is the most interesting—“data science”, “big + data”, and “social media” aren’t really great fits for this + study). + + In general, I’m excited by the ideas and motivation of this work. I + would be more positive if the data analysis process were more transparent + and clear throughout, and if the authors could make a more focused + contribution without relying on (what I would argue are) overused popular + terms. + +Rebuttal response + + + +------------------------ Submission 2826, Review 2 ------------------------ + +Reviewer: external + +Expertise + + 3 (Knowledgeable) + +Recommendation + + Strong Accept: I would argue strongly for accepting this paper; 5.0 + +Award Nomination + + If accepted, this paper would be among the top 5% of papers presented at CHI (Best Paper nomination) + +Your Assessment of this Paper's Contribution to HCI + + This paper looks at youth who used Scratch "blocks" to access metadata on + other users. The programs they created led the users to think deeply + about issues with Big Data. This study provides an understanding of 1) + youth experiences of Big Data (which has largely been understudied) 2) an + exploration of a system where users both produce and analyze metadata 3) + pedagogical suggestions for teaching data science. + +The Review + + What I liked: + + - There was a lot to like about this paper! It was surprising to me that + the issues with Big Data that scholars have noted were so readily + identified by youth on Scratch Community Blocks. The insights about how + children and teens view issues of Big Data were in line with things that + researchers have noted in their own critiques, but some of these issues + were presented in a new light when viewed through youths' perspectives. + + - I was initially skeptical about how such simple changes to Scratch + could provoke so much reflection from users, but I was thoroughly + convinced by the end of the paper that even relatively small changes to + Scratch had a significant impact on how users understood their + relationship to their data and the ethics of algorithms. + + - I was impressed with the rather comprehensive methods used by the + researchers. + + - The authors made strong arguments for using these sorts of techniques + for teaching about the ethics of data science. + + + Issues: + + - I was not clear on the importance of discussing "critical data fluency" + because it was not returned to explicitly later in the paper. The authors + seemed to position this concept as a contribution to discussions about + literacy, so I expected to see more about it. + + - The authors say "we focus on what can be termed as critical data + literacies" and then say that "the themes that we find are better + described as critical data fluencies rather than critical data + literacies". Which is the focus? + + - The discussion towards the end of the paper made me wonder whether the + users discuss ethical issues around data use and Big Data in terms of any + other platforms at any point. Did this experience with Scratch cause them + to reflect on these issues on other platforms they use (e.g., Instagram)? + If there is any data on this, then the authors may want to discuss this + in a revision. + + - The "data comes with assumptions and hidden decisions" section could + have been written more clearly. It was initially unclear how the project + described in this section worked and thus the reactions to this project + did not make complete sense to me, nor was it clear what the assumptions + or hidden decisions were. It was only when the authors later explained + that the algorithm that calculated the number of views was not visible to + the users that I understood that "the "assumptions and hidden decisions" + was not referring to those of the creator of the project, but rather, + those of the developers of these features of Scratch. + + - I felt that the quote from Burner (pg. 10) was a bit long and could + have been integrated into the paper better. + + - There were a few small grammar mistakes: + page 4: "during the days weeks" + page 7: "more subtler" + page 9: "quantifiable test scores gets" + page 10: "Our findings presents" + + + Summary: + + Significance of the paper's contribution to HCI and the benefit that + others can gain from the contribution: this research suggests unique + pedagogical methods for teaching data science and promoting literacy. It + also provides a deeper understanding of how users experience Big Data, + which is sometimes lacking in discussions about the ethics of Big Data + given the difficulty of researching this topic empirically. Lastly, it + explores the experiences of users who are both producers and analysts of + data--this seems like a relatively unique position for users to be in, + and I think creating more systems like this in the future may reveal new + ways of distributing power among actors in systems that use Big Data. + + Originality of the work: this work describes youth views on Big Data and + how they engage with data science. As far as I am aware, there is little + or no research on this topic. + + Validity of the work presented: I have confidence in the validity of + these findings, as the authors used many multiple methods to triangulate + their findings. + + Presentation clarity: it was very well written and organized in a logical + way that was easy to follow. + + Relevant previous work: I cannot comment on whether the authors + referenced all of the relevant literature in education or youth and + social media, but as for literature on Big Data, this seemed complete. + +Rebuttal response + + + +------------------------ Submission 2826, Review 3 ------------------------ + +Reviewer: external + +Expertise + + 2 (Passing Knowledge) + +Recommendation + + Neutral: I am unable to argue for accepting or rejecting this paper; 3.0 + +Award Nomination + + + +Your Assessment of this Paper's Contribution to HCI + + This works contributes to an understanding of how youth discuss and view + public data analytics of their social media interactions through the lens + of their use of Scratch Community Blocks. This work compares themes that + arise within its findings with existing discourses on Big Data. Finally, + the paper provides recommendations and implications for educators and + designers of learning environments. + +The Review + + Understanding children’s perspectives- their concerns and their ideas- + about how their personal social-media data is used in public, + particularly as it relates to Big Data uses, is a critical perspective + and a valuable contribution to HCI. The foundations in and discussions on + designs for educational technologies will be of value to HCI community + members who focus on developing learning technologies. + + Much of the writing in the work is strong. I particularly appreciate how + clear the authors were in their description of their role in the + research. However, there are structural issues that confuse the document. + + + The narrative of the paper is disjoint, putting a burden on the reader to + connect the ideas that are presented in the work. The introduction frames + the work in terms of Big Data: how it is used to analyze youth + interactions without allowing youths a voice and opinion on the process. + The framing presented around “highlighting youth’s voices in the + broader conversations of Big Data” is focused on a question the + Introduction and Abstract pose: What should young people know about the + data being collect about them and about the attempts to analyze and + understand these data in ways that can shape their experience? Yet it is + in the background that we discover that the paper will contribute to the + scholarly dialog on what data science education for youth may look like. + The discussion is, in fact, largely focused on recommendations and + implications for educators and designers of learning environments. While + certainly valuable to HCI practitioners who focus on educational + technologies, making it clear how the themes about big data in the + findings and participant perspectives on these themes influenced the + recommendations that were put forth would strengthen and unify the work. + Similarly, returning to the original position of the work, what *should* + young people know about the data being collected about them? + + Regarding another structural decision: While choosing to relate each of + the 5 themes in the findings to related work within their descriptions is + somewhat unorthodox within CHI papers, it largely worked. A minor point + is that the paper could do more to set reader’s expectations on this. + (For instance, it was only on my second read of the paper that I realized + the Background section’s introduction was supposed to initially set + this expectation— that text could be more explicit.) A more substantial + point is that the positioning of this work to similar veins of research + (rather than the individual findings) could be improved. + +Rebuttal response + + + + diff --git a/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-revision_summary.txt b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-revision_summary.txt new file mode 100644 index 0000000..c4b0eec --- /dev/null +++ b/chi_rebuttals/2017-critical_data_literacies/critical_data_literarices-CHI2017-revision_summary.txt @@ -0,0 +1,25 @@ +* We have changed the title of the paper to "Youth Perspectives on Critical Data Literacies" (as per R1's response to our rebuttal and as discussed in our letter to 1AC) + +* We have e the introduction and framing to focus on critical data literacies rather than social media, data science etc. As a result of this change, we have standardized on the term literacies throughout. The term "fluency" is no longer even mentioned in the manuscript. + +* We have added and expanded sections about our grounded theory methodology. We have reworked much of our "Data and Methodology" section to add text to make our methodology more clear. In particular, we have made it clear that we closely followed Charmaz's influential textbook on grounded theory which is different, in some respects, from Corbin and Strauss' seminal text. We have also made it clear that we used a single codebook and that we coded several "earlier" sources of data more openly than data (like the survey) that came in later. + +* In particular, we have expanded our discussion on our use of deductive codes and the process by which we derived them. As we suggested in our rebuttal, we used both "sensitizing" codes drawn from theory during initial coding as well as later-stage iterative recoding of data using codes informed by theory relevant to emergent themes. + +* Also, to better illustrate of the process by which we arrived at the themes we describe in our work, we added a paragraph to our data and methodology section that provides examples of codes we derived from open coding and themes which emerged from the codes. We hope that this makes our process and methodology much more transparent. + +* We added a clear outline of the rest of the paper to the end of our introduction.. + +* We have tried to set expectations about the structure of the findings subsections more explicitly. As per R3, we have added a sentence to the paragraph laying out our plan for the paper to explain to readers that we structure our findings so that we relate each one to prior work. + +* We have edited our discussion of generalizability in our introduction to make the limitations, in this regard, more clear. We have also added a brief discussion of how we do not know how kids might translate their insights into Scratch to other social media platforms. + +* We have a number of new new quotes to the paper's findings section and concluded the section on each literacy with a paragraph that explicitly ties it to existing theory and reiterates the connection to critical data literacies. + +* We have clarified the data comes with assumptions and hidden decisions sections as requested + +* We have shortened and integrated the quote from Bruner in our discussion. + +* We have shortened our discussion and conclusion and merged them into a single section. + +* We have fixed all the grammar mistakes that were pointed out by the reviewers.