mgaughan/covid19
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
master
covid19/cron-wikipedia_views.sh

20 lines
789 B
Bash
Raw Permalink Normal View History

add two small shellscripts for automation - Added two bash scripts usable as cronjobs to automate the production of revisions and view data. These commands automate the process of running code and copying material
2020-04-01 20:15:11 +00:00
#!/bin/bash -x
updated script to ensure the correct working dir
2020-04-07 21:39:58 +00:00
WORKING_DIR="/home/SOC.NORTHWESTERN.EDU/bmh1867/covid19"
cd $WORKING_DIR
add two small shellscripts for automation - Added two bash scripts usable as cronjobs to automate the production of revisions and view data. These commands automate the process of running code and copying material
2020-04-01 20:15:11 +00:00
TZ="UTC"
changes to allow historical view data collection - fix bug where it would fail if the first essay had no view data - add ability to override dates in the cron script
2020-04-02 18:28:34 +00:00
date_string=${OVERRIDE_DATE_STRING:-$(date +%Y%m%d)}
add two small shellscripts for automation - Added two bash scripts usable as cronjobs to automate the production of revisions and view data. These commands automate the process of running code and copying material
2020-04-01 20:15:11 +00:00
updated to just write a single log file for each day
2020-04-02 17:48:19 +00:00
view_log="enwp-daily_views-${date_string}.log"
changes to allow historical view data collection - fix bug where it would fail if the first essay had no view data - add ability to override dates in the cron script
2020-04-02 18:28:34 +00:00
./wikipedia/scripts/wikiproject_scraper.py 2> >(tee wikipedia/logs/${view_log})
add two small shellscripts for automation - Added two bash scripts usable as cronjobs to automate the production of revisions and view data. These commands automate the process of running code and copying material
2020-04-01 20:15:11 +00:00
# get the list of files
changes to allow historical view data collection - fix bug where it would fail if the first essay had no view data - add ability to override dates in the cron script
2020-04-02 18:28:34 +00:00
./wikipedia/scripts/fetch_enwiki_daily_views.py -d "${date_string}" 2> >(tee -a wikipedia/logs/${view_log})
change copy to move in cron scripts
2020-04-01 20:49:02 +00:00
mv wikipedia/logs/${view_log} /var/www/covid19/wikipedia/logs/${view_log}
mv wikipedia/data/digobs_covid19-wikipedia-enwiki_dailyviews-${date_string}.tsv /var/www/covid19/wikipedia/
add two small shellscripts for automation - Added two bash scripts usable as cronjobs to automate the production of revisions and view data. These commands automate the process of running code and copying material
2020-04-01 20:15:11 +00:00
# xz wikipedia/data/digobs_covid19-wikipedia-enwiki_dailyviews-${date_string}.json
change copy to move in cron scripts
2020-04-01 20:49:02 +00:00
mv wikipedia/data/digobs_covid19-wikipedia-enwiki_dailyviews-${date_string}.json /var/www/covid19/wikipedia/
add two small shellscripts for automation - Added two bash scripts usable as cronjobs to automate the production of revisions and view data. These commands automate the process of running code and copying material
2020-04-01 20:15:11 +00:00
Reference in New Issue Copy Permalink