Apache Airflow

You are an Airflow engineer. Produce a ready-to-deploy Airflow 2.x DAG (single Python file) that runs nightly to copy new CSV files from a specified S3 prefix into Snowflake. Constraints: use SnowflakeOperator or SnowflakeHook patterns, include S3 list/download step with AWS connection id, idempotent behavior (skip already-loaded files), 3 retries with exponential backoff, and clear task names. Output format: provide only the Python DAG file content with necessary imports, default_args, connections as variables, and brief inline comments. Example: schedule_interval '@daily', start_date two days ago.

You are an Airflow DAG author. Generate a concise Airflow 2.x DAG that schedules daily model training: data extraction, feature engineering, model training, evaluation, and artifact upload to S3. Constraints: use PythonOperator or KubernetesPodOperator placeholders, accept a run_date DAG parameter, fail if evaluation metric AUC < 0.75, and push the trained model path via XCom. Output format: return a single Python DAG file content with clear task ids, retry policy, parameter parsing, and small inline comments. Example: include a simple Python callable stub for 'train_model' that returns a file path.

You are a platform engineer designing CI for Airflow DAGs. Provide a structured CI pipeline (YAML steps) for GitHub Actions or GitLab CI that lints, unit-tests, packages, and deploys DAGs to an Airflow environment. Constraints: include flake8/ruff linting, pytest unit tests with an Airflow DAG import smoke-test, a build step producing a tarball artifact, and a safe deploy step that validates DAG file checksum and uploads to a target S3/GCS DAGs bucket or invokes provider API. Output format: YAML pipeline with named steps, shell commands, environment variables, and rollback guard (dry-run validation).

You are an SRE building SLA enforcement for Airflow. Create a concise plan and Airflow configuration snippet that enforces SLAs for critical DAG runs with email and PagerDuty alerts. Constraints: use Airflow SLA miss callbacks, set SLA per task, include exponential retry policy and alert deduplication window, and show sample integration with SMTP and PagerDuty webhook notification. Output format: provide (1) a YAML/INI snippet for airflow.cfg or secrets needed, (2) a Python SLA callback function, and (3) an example DAG task decorator applying the SLA with a short explanation of dedup logic.

You are a senior data platform engineer. Provide a detailed, multi-step optimization plan and a sample Airflow 2.x DAG pattern to process 2+ TB/day ETL across object storage and Snowflake. Tasks: (1) propose operator choices (e.g., partitioned COPY, multiprocessing, KubernetesPodOperator), (2) recommend executor, scheduler and worker sizing, pools, concurrency, and partitioning strategy, (3) include a code pattern for dynamic task mapping/parallelism with chunking and idempotent checkpoints, (4) provide metrics to monitor and expected resource estimates. Output format: (A) a one-paragraph architecture summary, (B) a Python DAG snippet demonstrating dynamic task mapping and pools, (C) a bullet list of monitoring metrics and numeric sizing heuristics.

You are an Airflow platform engineer building a dynamic DAG generation system. Produce a complete design and code examples to: (1) generate DAGs at runtime from a JSON config store, (2) use TaskGroups and dynamic task mapping for variable-length steps, (3) pass metadata with XComs safely (avoid large payloads), (4) include unit tests (pytest) for DAG integrity and a Git hook that prevents breaking changes. Output format: (A) short design doc (5-8 bullets), (B) Python code: generator function, one example generated DAG, XCom usage pattern, and a pytest example, (C) a sample pre-commit hook command.

Compare Apache Airflow with Prefect, Dagster, Luigi. Choose based on workflow fit, pricing, integrations, output quality and governance needs.