eae5464fd2
monitor pages from dsaez's wikidata crawler
2020-04-04 15:23:33 -07:00
cfe21254d9
rename scripts
2020-04-04 15:23:00 -07:00
c97028fabb
update cron scripts with new data format
2020-04-04 15:20:34 -07:00
974dc48b12
Merge branch 'master' of github.com:CommunityDataScienceCollective/COVID-19_Digital_Observatory into dsaez_submodule
2020-04-03 15:34:16 -07:00
a3e40a072f
Add dsaez's submodule for crawling wikidata
2020-04-03 15:24:56 -07:00
152704df7c
Merge pull request #17 from makoshark/master
...
changes to support historical view data
2020-04-02 14:18:05 -07:00
13371fd83e
Merge pull request #16 from aaronshaw/master
...
renames files, adds an example analysis w small sample of revisions data
2020-04-02 14:16:57 -07:00
aaronshaw
40f528f4ff
revisions to reflect updated example filename and clean comments in R code
2020-04-02 13:38:32 -05:00
Benjamin Mako Hill
1cec120dfa
changes to allow historical view data collection
...
- fix bug where it would fail if the first essay had no view data
- add ability to override dates in the cron script
2020-04-02 13:28:34 -05:00
Benjamin Mako Hill
3d0d4eee76
updated to just write a single log file for each day
2020-04-02 12:48:19 -05:00
aaronshaw
ba21acdf37
initial commit of revisions analysis example with output files
2020-04-02 10:59:44 -05:00
aaronshaw
4ab432f399
removing outdated file names
2020-04-02 07:49:08 -05:00
f770ade87a
Merge pull request #15 from aaronshaw/master
...
renaming example analysis directories
2020-04-01 19:15:21 -05:00
aaronshaw
576d882c04
renaming example analysis directories
2020-04-01 19:12:45 -05:00
groceryheist
ff96d52cb9
Merge pull request #12 from makoshark/master
...
substantial changes to wikipedia fetching code
2020-04-01 16:36:56 -07:00
Benjamin Mako Hill
ff5521d44b
ignore __pycache__
2020-04-01 18:23:50 -05:00
Benjamin Mako Hill
b26a2b5a86
fix bug in previous commit
...
forgot to import digobs module in the scraper script
2020-04-01 18:22:36 -05:00
Benjamin Mako Hill
427eddd141
cleaned up unnecessary files
2020-04-01 18:21:41 -05:00
Benjamin Mako Hill
b457cd726b
use the type= feature in argparse
...
- integrated the type= feature in argparse in all three scripts
- removed some redundant code from the third file
2020-04-01 18:13:02 -05:00
Benjamin Mako Hill
17c3f75389
Merge branch 'master' of github.com:CommunityDataScienceCollective/COVID-19_Digital_Observatory
2020-04-01 17:19:33 -05:00
Benjamin Mako Hill
070d23f718
changes in response to code review by nate
...
- moved some common functions into files
- other smaller changes
2020-04-01 17:16:34 -05:00
34f8b9a23e
Merge pull request #14 from aaronshaw/aaronshaw-master
...
pointing at updated data url, adding explicit NA handling to factor, …
2020-04-01 16:58:02 -05:00
aaronshaw
282588772e
pointing at updated data url, adding explicit NA handling to factor, cutting unnecessary call to ggplot2, and updated corresponding output from new data file. May not work while kibo urls are getting resolved
2020-04-01 16:52:22 -05:00
Benjamin Mako Hill
4fe5deb013
Merge branch 'master' of github.com:CommunityDataScienceCollective/COVID-19_Digital_Observatory
2020-04-01 16:42:16 -05:00
Benjamin Mako Hill
d655e1ce93
tweaks to revision export code
...
- flags were not being exported (e.g., minor, anon)
- broke with hidden/deleted user names
2020-04-01 16:39:53 -05:00
Benjamin Mako Hill
3f19805d36
fix bug in rev scraper script
...
Bug was a break, added for debugging, that caused the script to only
work for the first article.
2020-04-01 15:49:28 -05:00
Benjamin Mako Hill
95d37cff7a
change copy to move in cron scripts
2020-04-01 15:49:02 -05:00
Benjamin Mako Hill
5739d1c404
Merge branch 'master' of github.com:makoshark/COVID-19_Digital_Observatory
2020-04-01 15:18:50 -05:00
Benjamin Mako Hill
141871eda6
add two small shellscripts for automation
...
- Added two bash scripts usable as cronjobs to automate the production
of revisions and view data.
These commands automate the process of running code and copying material
2020-04-01 15:15:11 -05:00
Benjamin Mako Hill
04e00f363b
address confusion with date
...
The timestamps in files should be the day that the exports are done. For
the view data, the query date needs to be the day before but this
shouldn't be the timestamp we use in files, etc.
2020-04-01 15:14:05 -05:00
Benjamin Mako Hill
06d2fd1563
fix bugs with the date stamps
2020-04-01 10:47:33 -05:00
4f8a698c62
Merge pull request #11 from jdfoote/master
...
Adding a tidyverse example (with very verbose comments)
2020-04-01 10:41:02 -05:00
Benjamin Mako Hill
4e1b7fbdfe
fixed typo in debug message
2020-04-01 08:18:05 -07:00
Benjamin Mako Hill
061105b7b4
Merge branch 'master' of github.com:CommunityDataScienceCollective/COVID-19_Digital_Observatory
2020-04-01 07:53:40 -07:00
Benjamin Mako Hill
268f9e1cf3
added gitignore for wikipedia/data directory
2020-04-01 07:52:15 -07:00
Benjamin Mako Hill
784458f206
renamed the wikipedia_views module to wikipedia
2020-04-01 07:51:20 -07:00
Benjamin Mako Hill
6493361fbd
added initial version of revision-scraper
...
Borrows much of the structure from the (patched) version of the
dailyview scraper.
2020-04-01 07:42:38 -07:00
Benjamin Mako Hill
cb26ecabda
fixed typo in description of view scraper
2020-04-01 07:42:24 -07:00
Benjamin Mako Hill
5c861cfca4
renamed daily views to make it clear that it's just enwiki
2020-04-01 07:29:45 -07:00
Benjamin Mako Hill
38fdd07b39
changes to a bunch of the wikipedia view code
...
- Renamed the articles.txt to something more specific
Changes to both scripts:
- Updated filenames to match the new standard
- Reworked the logging code so that it can write to stderr by
default. Because we can only call logging.basicConfig() once, this
eneded up being a bigger changes.
- Caused scripts to output git commits and export to track which code
produced which dataset.
- Caused programs to take files instead of directories as
output (allows us to run programs more than once a day).
Changes to the wikipedia_views/scripts/fetch_daily_views.py:
- Change output that it outputs a sequence of JSON dictionaries (one
per line) as per the standard we agreed to and which is what
Twitter, Github, and other dumps do. Previous behavior was to create
output a single JSON list object.
- A number of other small changes and tweaks throughout.
2020-04-01 07:15:12 -07:00
Jeremy Foote
6b05896aa5
Adding a tidyverse example (with very verbose comments)
2020-03-31 22:42:31 -04:00
8bb3db8b46
add examples using the translations data
2020-03-31 16:56:59 -07:00
c8b886364f
add documentation for the output files
2020-03-31 16:22:30 -07:00
29ae62c83e
create 'latest.csv' to link to the most recent output.
2020-03-31 16:16:36 -07:00
687da1284f
Merge branch 'master' of github.com:CommunityDataScienceCollective/COVID-19_Digital_Observatory
2020-03-31 16:01:43 -07:00
603a7b6ec3
update output
2020-03-31 16:01:38 -07:00
74667cf4dc
use 'item' instead of 'entity'
2020-03-31 15:34:34 -07:00
3d142377ca
rename compile script
2020-03-31 15:27:39 -07:00
55110c7f21
update compile script
2020-03-31 15:27:21 -07:00
4fd516a700
Improve README.md for keywords
2020-03-31 15:25:51 -07:00
