collective/covid19
18
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

changes to allow historical view data collection

Browse Source 
- fix bug where it would fail if the first essay had no view data
- add ability to override dates in the cron script
This commit is contained in:
Benjamin Mako Hill
2020-04-02 13:28:34 -05:00
parent 3d0d4eee76
commit 1cec120dfa
2 changed files with 6 additions and 5 deletions
Download Patch File Download Diff File Expand all files Collapse all files

6
cron-wikipedia_views.sh
View File

@@ -1,13 +1,13 @@
#!/bin/bash -x
TZ="UTC"
date_string=$(date +%Y%m%d)
date_string=${OVERRIDE_DATE_STRING:-$(date +%Y%m%d)}
view_log="enwp-daily_views-${date_string}.log"
./wikipedia/scripts/wikiproject_scraper.py 2> >(tee wikipedia/logs/{$view_log})
./wikipedia/scripts/wikiproject_scraper.py 2> >(tee wikipedia/logs/${view_log})
# get the list of files
./wikipedia/scripts/fetch_enwiki_daily_views.py 2> >(tee -a wikipedia/logs/${view_log})
./wikipedia/scripts/fetch_enwiki_daily_views.py -d "${date_string}" 2> >(tee -a wikipedia/logs/${view_log})
mv wikipedia/logs/${view_log} /var/www/covid19/wikipedia/logs/${view_log}
mv wikipedia/data/digobs_covid19-wikipedia-enwiki_dailyviews-${date_string}.tsv /var/www/covid19/wikipedia/