Write downloader task for airflow #3
Description
We need to set up a periodic airflow task to do the following roughly every hour:
- access U of Iowa archive of MRMS data
- scan today's directory structure for list of data files
- check previously downloaded files for difference
- if different or new, download file
- signal airflow about changes (if necessary)
On backfill,
- download with at least 20 seconds between files
Questions:
- what signals can we derive to detect file changes (perhaps if partially complete files are posted)?
- how should we test the data integrity? Size? Quick format check by reading grib file?
Links:
See https://github.com/agstack/weather-server/blob/main/experiments/data/mrms.jl for data URLs