collective/coldcallbot
18
0
Fork 0
Code Activity

rearrange the repository for publication

Browse Source
This commit is contained in:
Benjamin Mako Hill
2020-12-29 12:22:35 -08:00
parent 743e0a39f3
commit 9c4f81c30a
5 changed files with 109 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

105
assessment_and_tracking/README Normal file
View File

@@ -0,0 +1,105 @@
I don't expect that these will necessary work without
modification. It's a good idea to go line-by-line through these to
make sure they are doing what *you* want and that you agree with the
assessment logic built into this.
Daily Process
======================================
After class, you will have two new files created that will be named
like this (with today's date):
attendance-2020-10-05.tsv
call_list-2020-10-05.tsv
Each day, you need to open up "call_list-YYYY-MM-DD.tsv" and edit the
final two columns. The first columns `answered` means that the person
responded and answered the question (i.e., they were present in the
room but away from their computer and unresponsive). This is almost
always TRUE but would be FALSE if the student were missing.
The final column `assessment` is GOOD, FAIR, or BAD in my rubric. I've
detailed what that means on this page:
https://wiki.communitydata.science/User:Benjamin_Mako_Hill/Assessment#Rubric_for_case_discussion_answers
I take notes on student answers on paper during class (typically I
only note down non "GOOD" answers) and then add these to the sheet
immediately after class.
I keep my entire data directory in git and I'd recommend that you do
too.
Other Notes
======================================
1.
These scripts rely on a file in this repository called
`data/student_information.csv` which I have set to be downloaded
automatically from a Google form using a 1-line wget command.
For reference, that file has the following column labels (this is the
full header, in order):
Timestamp
Your UW student number
Name you'd like to go by in class
Your Wikipedia username
Your username on the class Discord server
Preferred pronouns
Anything else you'd like me to know?
2.
The scripts in this directory are meant to be run or sourced *from*
the data directory. As in:
$ cd ../data
$ R --no-save < ../assessment_and_tracking/track_participation.R
3.
There are three files here:
track_enrolled.R:
This file keeps track of who is in Discord, who is enrolled for
the class, etc. This helps me remove people from the
student_informaiton.csv spreadsheet who are have dropped the
class, deal with users who change their Discord name, and other
things that the scripts can't deal with automatically.
This all need to be dealt with manually, one way or
another. Sometimes by modifying the script, sometimes by modifying
the files in the data/ directory.
This requires an additional file called
`myuw-COM_482_A_autumn_2020_students.csv` which is just the saved
CSV from https://my.uw.edu which includes the full class list. I
download this one manually.
track_participation.R:
This file generates histograms and other basic information about
the distribution of participation and absences. I've typically run
this weekly after a few weeks of the class and share these images
with students at least once or twice in the quarter.
This file is also sourced by compute_final_case_grades.R.
compute_final_case_grades.R:
You can find a narrative summary of my assessment process here:
https://wiki.communitydata.science/User:Benjamin_Mako_Hill/Assessment#Overall_case_discussion_grade
This also requires the registration file (something like
`myuw-COM_482_A_autumn_2020_students.csv`) which is described
above.
4.
A bunch of things in these scripts assumes a UW 4.0 grade scale. I
don't think it should be hard to map these onto some other scale, but
that's an exercise I'll leave up to those that want to do this.

119
assessment_and_tracking/compute_final_case_grades.R Normal file
View File

@@ -0,0 +1,119 @@
## load in the data
#################################
case.sessions <- 15
myuw <- read.csv("myuw-COM_482_A_autumn_2020_students.csv", stringsAsFactors=FALSE)
## class-level variables
question.grades <- c("GOOD"=100, "FAIR"=100-(50/3.3), "BAD"=100-(50/(3.3)*2))
missed.question.penalty <- (50/3.3) * 0.2 ## 1/5 of a full point on the GPA scale
source("../assessment_and_tracking/track_participation.R")
setwd("case_grades")
rownames(d) <- d$discord.name
## show the distribution of assessments
table(call.list.full$assessment)
prop.table(table(call.list.full$assessment))
table(call.list.full$answered)
prop.table(table(call.list.full$answered))
total.questions.asked <- nrow(call.list.full)
## create new column with number of questions present
d$prop.asked <- d$num.calls / d$num.present
## generate statistics using these new variables
prop.asks.quantiles <- quantile(d$prop.asked, probs=seq(0,1, 0.01))
prop.asks.quantiles <- prop.asks.quantiles[!duplicated(prop.asks.quantiles)]
## this is generating broken stuff but it's not used for anything
d$prop.asked.quant <- cut(d$prop.asked, breaks=prop.asks.quantiles,
labels=names(prop.asks.quantiles)[1:(length(prop.asks.quantiles)-1)])
## generate grades
##########################################################
d$part.grade <- NA
## print the median number of questions for (a) everybody and (b)
## people that have been present 75% of the time
median(d$num.calls[d$days.absent < 0.25*case.sessions])
median(d$num.calls)
questions.cutoff <- median(d$num.calls)
## helper function to generate average grade minus number of missing
gen.part.grade <- function (x.discord.name) {
q.scores <- question.grades[call.list$assessment[call.list$discord.name == x.discord.name]]
base.score <- mean(q.scores, na.rm=TRUE)
## number of missing days
missing.days <- nrow(missing.in.class[missing.in.class$discord.name == x.discord.name,])
## return the final score
data.frame(discord.name=x.discord.name,
part.grade=(base.score - missing.days * missed.question.penalty))
}
tmp <- do.call("rbind", lapply(d$discord.name[d$num.calls >= questions.cutoff], gen.part.grade))
d[as.character(tmp$discord.name), "part.grade"] <- tmp$part.grade
## next handle the folks *under* the median
## first we handle the zeros
## step 1: first double check the people who have zeros and ensure that they didn't "just" get unlucky"
d[d$num.calls == 0,]
## set those people to 0 :(
d$part.grade[d$num.calls == 0] <- 0
## step 2 is to handle folks who got unlucky in the normal way
tmp <- do.call("rbind", lapply(d$discord.name[is.na(d$part.grade) & d$prop.asked <= median(d$prop.asked)], gen.part.grade))
d[as.character(tmp$discord.name), "part.grade"] <- tmp$part.grade
## the people who are left are lucky and still undercounted so we'll penalize them
d[is.na(d$part.grade),]
penalized.discord.names <- d$discord.name[is.na(d$part.grade)]
## generate the baseline participation grades as per the process above
tmp <- do.call("rbind", lapply(penalized.discord.names, gen.part.grade))
d[as.character(tmp$discord.name), "part.grade"] <- tmp$part.grade
## now add "zeros" for every questions that is below the normal
d[as.character(penalized.discord.names),"part.grade"] <- ((
(questions.cutoff - d[as.character(penalized.discord.names),"num.calls"] * 0) +
(d[as.character(penalized.discord.names),"num.calls"] * d[as.character(penalized.discord.names),"part.grade"]) )
/ questions.cutoff)
d[as.character(penalized.discord.names),]
## map part grades back to 4.0 letter scale and points
d$part.4point <-round((d$part.grade / (50/3.3)) - 2.6, 2)
d[sort.list(d$prop.asked), c("discord.name", "num.calls", "num.present",
"prop.asked", "prop.asked.quant", "part.grade", "part.4point",
"days.absent")]
d[sort.list(d$part.4point), c("discord.name", "num.calls", "num.present",
"prop.asked", "prop.asked.quant", "part.grade", "part.4point",
"days.absent")]
## writing out data
quantile(d$num.calls, probs=(0:100*0.01))
d.print <- merge(d, myuw[,c("StudentNo", "FirstName", "LastName", "UWNetID")],
by.x="student.num", by.y="StudentNo")
write.csv(d.print, file="final_participation_grades.csv")
library(rmarkdown)
for (x.discord.name in d$discord.name) {
render(input="../../assessment_and_tracking/student_report_template.Rmd",
output_format="html_document",
output_file=paste("../data/case_grades/student_reports/",
d.print$UWNetID[d.print$discord.name == x.discord.name],
sep=""))
}

25
assessment_and_tracking/student_report_template.Rmd Normal file
View File

@@ -0,0 +1,25 @@
**Student Name:** `r paste(d.print[d.print$discord.name == x.discord.name, c("FirstName", "LastName")])`
**Discord Name:** `r d.print[d.print$discord.name == x.discord.name, c("discord.name")]`
**Participation grade:** `r d.print$part.4point[d.print$discord.name == x.discord.name]`
**Questions asked:** `r d.print[d$discord.name == x.discord.name, "prev.questions"]`
**Days Absent:** `r d.print[d.print$discord.name == x.discord.name, "days.absent"]` / `r case.sessions`
**List of questions:**
```{r echo=FALSE}
call.list[call.list$discord.name == x.discord.name,]
```
**Luckiness:** `r d.print[d.print$discord.name == x.discord.name, "prop.asked.quant"]`
If you a student has a luckiness over 50% that means that they were helped by the weighting of the system and/or got lucky. We did not penalize *any* students with a luckiness under 50% for absences.

23
assessment_and_tracking/track_enrolled.R Normal file
View File

@@ -0,0 +1,23 @@
myuw <- read.csv("myuw-COM_482_A_autumn_2020_students.csv")
gs <- read.delim("student_information.tsv")
## these are students who dropped the class (should be empty)
gs[!gs$Your.UW.student.number %in% myuw$StudentNo,]
## these are students who are in the class but didn't reply to the form
myuw[!myuw$StudentNo %in% gs$Your.UW.student.number,]
## read all the folks who have been called and see who is missing from
## the google sheet
call.list <- unlist(lapply(list.files(".", pattern="^attendance-.*tsv$"), function (x) {
d <- read.delim(x)
strsplit(d[[2]], ",")
})
)
present <- unique(call.list)
present[!present %in% gs[["Your.username.on.the.class.Discord.server"]]]
## and never attended class..
gs[["Your.username.on.the.class.Discord.server"]][!gs[["Your.username.on.the.class.Discord.server"]] %in% present]

109
assessment_and_tracking/track_participation.R Normal file
View File

@@ -0,0 +1,109 @@
library(ggplot2)
library(data.table)
gs <- read.delim("student_information.tsv")
d <- gs[,c(2,5)]
colnames(d) <- c("student.num", "discord.name")
call.list <- do.call("rbind", lapply(list.files(".", pattern="^call_list-.*tsv$"), function (x) {read.delim(x)[,1:4]}))
colnames(call.list) <- gsub("_", ".", colnames(call.list))
call.list$day <- as.Date(call.list$timestamp)
## drop calls where the person wasn't present
call.list.full <- call.list
call.list[!call.list$answered,]
call.list <- call.list[call.list$answered,]
call.counts <- data.frame(table(call.list$discord.name))
colnames(call.counts) <- c("discord.name", "num.calls")
d <- merge(d, call.counts, all.x=TRUE, all.y=TRUE, by="discord.name"); d
## set anything that's missing to zero
d$num.calls[is.na(d$num.calls)] <- 0
attendance <- unlist(lapply(list.files(".", pattern="^attendance-.*tsv$"), function (x) {d <- read.delim(x); strsplit(d[[2]], ",")}))
file.to.attendance.list <- function (x) {
tmp <- read.delim(x)
d.out <- data.frame(discord.name=unlist(strsplit(tmp[[2]], ",")))
d.out$day <- rep(as.Date(tmp[[1]][1]), nrow(d.out))
return(d.out)
}
attendance <- do.call("rbind",
lapply(list.files(".", pattern="^attendance-.*tsv$"),
file.to.attendance.list))
## create list of folks who are missing in class
missing.in.class <- call.list.full[is.na(call.list.full$answered) |
(!is.na(call.list.full$answered) & !call.list.full$answered),
c("discord.name", "day")]
missing.in.class <- unique(missing.in.class)
setDT(attendance)
setkey(attendance, discord.name, day)
setDT(missing.in.class)
setkey(missing.in.class, discord.name, day)
## drop presence for people on missing days
attendance[missing.in.class,]
attendance <- as.data.frame(attendance[!missing.in.class,])
attendance.counts <- data.frame(table(attendance$discord.name))
colnames(attendance.counts) <- c("discord.name", "num.present")
d <- merge(d, attendance.counts,
all.x=TRUE, all.y=TRUE,
by="discord.name")
days.list <- lapply(unique(attendance$day), function (day) {
day.total <- table(call.list.full$day == day)[["TRUE"]]
lapply(d$discord.name, function (discord.name) {
num.present <- nrow(attendance[attendance$day == day & attendance$discord.name == discord.name,])
if (num.present/day.total > 1) {print(day)}
data.frame(discord.name=discord.name,
days.present=(num.present/day.total))
})
})
days.tmp <- do.call("rbind", lapply(days.list, function (x) do.call("rbind", x)))
days.tbl <- tapply(days.tmp$days.present, days.tmp$discord.name, sum)
attendance.days <- data.frame(discord.name=names(days.tbl),
days.present=days.tbl,
days.absent=length(list.files(".", pattern="^attendance-.*tsv$"))-days.tbl)
d <- merge(d, attendance.days,
all.x=TRUE, all.y=TRUE, by="discord.name")
d[sort.list(d$days.absent), c("discord.name", "num.calls", "days.absent")]
## make some visualizations of whose here/not here
#######################################################
png("questions_absence_histogram_combined.png", units="px", width=800, height=600)
ggplot(d) +
aes(x=as.factor(num.calls), fill=days.absent, group=days.absent) +
geom_bar(color="black") +
scale_x_discrete("Number of questions asked") +
scale_y_continuous("Number of students") +
scale_fill_continuous("Days absent", low="red", high="blue")+
theme_bw()
dev.off()
png("questions_absenses_boxplots.png", units="px", width=800, height=600)
ggplot(data=d) +
aes(x=as.factor(num.calls), y=days.absent) +
geom_boxplot() +
scale_x_discrete("Number of questions asked") +
scale_y_continuous("Days absent")
dev.off()