Pipedream

Role: You are a backend engineer building a Pipedream HTTP-trigger workflow. Constraints: produce a single Node.js handler (CommonJS) that parses JSON payloads, validates required fields (id, email, name), and performs an upsert into Postgres using parameterized queries; use process.env for secrets named PG_HOST, PG_USER, PG_PASS, PG_DB; ensure idempotency by using an event_id header. Output format: provide full runnable code, brief SQL CREATE TABLE schema, and the exact environment variable names. Example webhook: {"id":"123","email":"[email protected]","name":"Alice"} and header X-Event-Id: evt_abc123.

Role: You are an SRE implementing a Pipedream scheduled workflow. Constraints: implement a Node.js step that requests a provided URL, marks failure if HTTP status >=500 or response time >500ms, retry the request up to 2 times with 300ms backoff, and POST a concise alert message to a Slack webhook stored in SECRET SLACK_WEBHOOK_URL. Output format: return a compact JSON report {url,status_code,response_time_ms,success,reason} and the exact Slack payload to send. Example input: {"url":" Provide runnable code suitable for a single Pipedream step.

Role: You are a data engineer designing a Pipedream scheduled ETL job. Constraints: support CSV URLs or S3 object paths, stream and chunk rows into batches of N (variable), deduplicate rows using an import_id column, and implement exponential backoff retries for transient failures (max 5 attempts). Provide code in Node.js that shows reading, mapping, and uploading to a target (e.g., Snowflake or BigQuery) via a placeholder client call; include a JSON mapping schema for CSV headers -> table columns and a config object {chunkSize, importId, maxRetries}. Output format: runnable pseudocode, mapping JSON example, and retry logic snippet. Example CSV header: id,email,created_at,amount.

Role: You are a product operations engineer building a Pipedream workflow step to validate and normalize webhooks before DB writes. Constraints: accept arbitrary webhook JSON, apply validation rules (required fields, email format, timestamp ISO8601), transform fields (camelCase -> snake_case), and output an array of parameterized SQL insert objects. Provide two example inputs and their normalized outputs. Output format: return JSON {rows:[{params:[v1,v2...], sql:'INSERT INTO ... VALUES ($1,$2...)'}], errors:[...]} and a short validation rules list. Examples: 1) single object payload, 2) nested payload with array of items.

Role: You are a senior backend engineer designing an idempotency/deduplication layer for Pipedream event pipelines. Multi-step: (1) propose architecture using Redis (or Pipedream-managed cache) with TTL-based locks and Lua script for atomic check-and-set, (2) provide Node.js code for generating idempotency keys from event headers/body, (3) give Redis commands/Lua script for atomic reserve-and-expire, (4) show retry/backoff policy and how to surface metrics (processed, duplicates, lock-fails) for alerts. Few-shot examples: show handling for event with id 'evt1' processed twice and a concurrent duplicate. Output format: textual architecture, Node.js snippets, Lua script, example Redis traces.

Role: You are an SRE/data engineer building an observability pipeline in Pipedream. Multi-step: ingest JSON telemetry from HTTP triggers and Kafka, normalize schemas, compute rolling-window metrics (1m, 5m error rate, p95 latency) in-code, write to a time-series DB (Influx/Prometheus remote/write), and emit alerts when thresholds are crossed with severity levels. Provide Python or Node.js snippets that compute sliding-window aggregates, a sample alert evaluation rule JSON, and a step-by-step Pipedream workflow mapping (triggers, transforms, storage, alerting). Few-shot examples: a burst of 5xxs leading to a high-severity alert, and a p95 latency spike creating a warning. Output format: plan, code snippets, alert rule examples.

Compare Pipedream with Zapier, n8n, Make (Integromat). Choose based on workflow fit, pricing limits, governance, integrations and how much human review is required.