Set up airflow #2
Description
We need a practice instance of airflow to try out ideas.
This should use the following config options:
- db should be postgres
- webserver port 8091
- set up cron job to run every 30 seconds to pull from github and copy dags into airflow dir atomically (pull, cp to tmp, mv)
- logging to file with rotation
Questions
We need to answer and document a few questions:
- can we have a single airflow instance run production/staging with different user ID to protect data?
- do we need multiple airflow instances launched as different IDs to separate production and staging?
- if we have multiple instances, can we limit resource consumption before running with K8S?
Links
http://airflow.apache.org/docs/apache-airflow/stable/security/workload.html