https://www.reddit.com/r/dataengineering/comments/s78jvx/best_job_scheduler_in_2022_airflow_dagster/
r/dataengineering on Reddit: Best job scheduler in 2022? (Airflow / Dagster / Prefect / Luigi / other)
Posted by u/UglyChihuahua - 56 votes and 33 comments
www.reddit.com
droplet에서 airflow는 db도 띄우고 생난리를 쳐야 해서 못 쓰겠더라. 램이 4gb밖에 안 되는데 airflow가 2gb는 먹고 들어간다.
yacron/cron, make를 생으로 쓰면서 맨날 에러 뜨니까 관리가 안 된다.
로깅이나 alert를 생으로 구현해야 하고, 정말 노답임.
https://dagster.io/blog/dagster-airflow
Dagster vs. Airflow | Dagster Blog
We often get asked why a data team should choose Dagster over Apache Airflow. We compare Dagster and Airflow for data orchestration, in five parts.
dagster.io
On the other hand, Dagster’s default execution is extremely lightweight. You can execute a job within a Python process by calling a function, without any persistent service up and running. If you want access to Dagster’s UI, you can spin it up with a single command and don’t need to run a database or connect to any services in the cloud. From Dagster’s UI, you can go to any job and click a button to immediately launch a run.
이런 게 있구만...
I'd argue not to use Airflow if you start fresh and use either:
Prefect if you need a fast and dynamic modern orchestration with a straightforward way to scale out. They recently revamped the prefect core as Prefect 2.0 with a new second-generation orchestration engine called Orion. It has several abstractions that make it a swiss army knife for general task management.
Dagster when you foresee higher-level data engineering problems. Dagster has more abstractions as they grew from first principles with a holistic view in mind from the very beginning. They focus heavily on data integrity, testing, idempotency, data assets, etc.
A good sense of what has changed between Airflow and newer tools, you see on Dagster vs. Airflow.
If you want to know more, I wrote about The Shift From Data Pipelines to Data Products.
https://docs.prefect.io/latest/getting-started/installation/
Installation - Prefect Docs
Installing Prefect and configuring on supported environments.
docs.prefect.io
Prefect도 좀 좋아보임.
PERfect가 아니라 PREfect인걸 주의해야 함
https://dagster.io/vs/dagster-vs-prefect
Dagster vs Prefect: how to compare the two
Dagster orchestration vs Prefect
dagster.io
서로 다른 일을 한다고 하는데 비교는 자주 된다고...
내가 하려는 일은 task를 하는거지 데이터 그 자체를 모으는 일은 아니니까 (db작업도 하긴 허는데) prefect를 일단 골라볼까나... decorator만 달면 되는 것으로 보인다....
Prefect는 써보니까 dask같은 느낌이 든다. 실제로 dask backend도 있고 parallel processing을 하려면 futures 사용도 해야됨.
dagster쪽으로 다시 알아봐야겠다.