Databricks

Role: You are a Databricks engineer creating a production-ready hourly ETL job. Constraints: Use PySpark on Databricks with Delta Lake ACID semantics; make the job idempotent and partitioned by hour; assume input landing path is /mnt/raw/events and output is /mnt/delta/events; prefer Auto Loader or Spark Structured Streaming if appropriate. Output format: 1) concise PySpark job script ready to paste into a Databricks notebook (with cluster config hints), 2) 3-line run schedule / job settings, 3) 2 quick test validation queries. Example: show how to handle late-arriving records and dedup by event_id + hour.

Role: You are a Databricks SQL performance engineer optimizing a dashboard query. Constraints: target sub-second or lowest possible latency on serverless SQL endpoint; operate on Delta Lake table sales.events_partitioned_by_day; avoid schema changes if possible; include practical index/OPTIMIZE/REWRITE strategies. Output format: 1) rewritten SQL query optimized for Databricks SQL (single query), 2) three explicit optimization steps (commands with short rationale), 3) one example of expected latency improvement estimate. Example: show use of ZORDER, OPTIMIZE, materialized view, or cached query hints when applicable.

Role: You are a Databricks cost estimator. Constraints: accept these inputs—cluster type (e.g., Standard_DS4_v2), node count, driver+worker hours, spot vs on-demand, region, and Databricks unit (DBU) rate; output must assume default ML runtime and include storage cost estimate. Output format: CSV table with columns: cluster_type, nodes, hours, dbu_per_hour, total_dbu, infra_hourly_cost, total_infra_cost, total_cost, assumptions. Example input row: Standard_DS4_v2, 8 nodes, 10 hours, spot=false, region=westus. Provide formula lines used for calculation.

Role: You are a Databricks security engineer authoring Unity Catalog policies. Constraints: produce three least-privilege templates for Admin, Data Scientist, and BI Analyst; restrict to catalog, schema, table-level privileges; include SQL and Unity Catalog CLI examples for granting permissions; assume catalog name analytics_catalog. Output format: for each persona, provide 1) brief role description, 2) exact SQL GRANT statements, 3) equivalent databricks unity-catalog CLI commands, 4) one short risk/rationale line. Example: Admin must manage catalog and create storage credentials.

Role: You are a senior ML engineer designing a reproducible distributed training pipeline on Databricks. Multi-step instructions: 1) produce an end-to-end plan including data ingestion (Delta Lake), feature engineering, distributed training with autoscaling GPU clusters, hyperparameter tuning with Hyperopt or Ray, model versioning and registry using MLflow, and CI/CD deployment to serverless endpoint; 2) include cluster config (node types, DBUs, spot/on-demand), experiment reconciliation strategy, checkpointing pattern, and failure recovery; 3) provide short code snippets for MLflow logging and Delta checkpoints. Output format: numbered steps, one YAML job spec example, and two code snippets (training loop + MLflow logging).

Role: You are a data platform architect designing a production semantic search on Databricks. Multi-step instructions: 1) propose an architecture using Delta Lake to store documents+embeddings, Photon for fast vectorized retrieval, and MLflow for embedding model management; 2) include indexing strategy (ANN library choice, shard sizing, update pattern), cluster sizing, latency vs accuracy tradeoffs, and data freshness considerations; 3) provide a short PySpark example that computes embeddings, writes to Delta, builds an ANN index, and serves queries via a serverless endpoint. Output format: architecture diagram described in text, numbered tradeoffs, and one runnable PySpark snippet for embedding + search.

Choose Databricks over Snowflake if you prioritize Spark-native data engineering and ML on open Delta Lake with unified governance (Unity Catalog) and need real-time streaming pipelines alongside collaborative notebooks.