Senior Data Engineer
Senior Data Engineer with production experience designing, optimizing, and operating large-scale Spark workloads on Microsoft Fabric or Databricks.
Strong in Delta Lake performance (liquid clustering, OPTIMIZE, file sizing, incremental merges), Spark UI-driven troubleshooting (skew, spill, OOM, shuffle), and Fabric capacity economics (CU-seconds, bursting, smoothing, throttling stages, pool and node sizing). Builds reliable, cost-aware pipelines that stay inside capacity limits while meeting SLAs for both interactive and batch workloads.
Also develops and maintains internal Python libraries and reusable Spark modules that standardize ingestion, governance, and optimization patterns across teams.
Core Technical Strengths
Spark load optimization & troubleshooting: Stagelevel diagnosis (task duration histograms, shuffle skew, spill), AQE, broadcast vs shuffle joins, predicate pushdown, salting only when proven, driver vs executor memory pressure, Python UDF cost.
Delta Lake & Fabric Lakehouse: Liquid clustering vs classic partitioning + Z-order, target file size, small-file mitigation, Vorder for Direct Lake, statistics, VACUUM retention vs time travel, idempotent incremental MERGE patterns with watermarks and quarantine.
Fabric capacity & compute: CU-second accounting, 10 min / 60 min / 24 h throttling windows (overage protection -> interactive delay -> interactive rejection -> background rejection), carryforward and burndown, starter vs custom vs custom live pools, node sizing (Small through XX-Large) and its effect on max node count, auto-scale and dynamic allocation, single node resource sharing, CU budgeting across concurrent jobs.
Operational design: Separating interactive, scheduled, and heavy batch workloads; hard caps on pool max nodes; capacity isolation strategies; monitoring via Capacity Metrics app and Spark resource usage UI; maintenance pipelines (OPTIMIZE timing, statistics).
Python library development: Designs, packages, and versions internal Python libraries and Spark utility modules (ingestion frameworks, Delta metadata/governance helpers, role and security managers, monitoring utilities). Emphasizes production readiness - type hints, logging, error handling, configuration-driven behavior, unit/integration testing, and clear READMEs - so notebooks stay thin and teams share consistent, maintainable code.
Working Style
Starts with measurement (Spark UI, Capacity Metrics, file-level Delta stats) before changing infrastructure. Prefers code and configuration changes that reduce shuffled bytes and future CU-second debt over "make the pool bigger."
Documents pool sizing rationale, throttling risks, and maintenance expectations so the next engineer can operate the system.
Comfortable explaining capacity behavior to both engineering and platform teams.
Built and iterated production Python libraries and notebook modules used across multiple Fabric workspaces, including standardized Delta ingestion classes (with governance metadata and TBLPROPERTIES support), OneLake data-access role management utilities, and reusable optimization/maintenance helpers. Focused on thread-safety, validation, secrets handling, and documentation so the libraries could be adopted by other engineers without constant support.