Capital Group
Senior Data Engineer
- Ingest from third-party REST APIs, Salesforce, and vendor flat-file drops — CSV, XML, JSON and Parquet — landing everything raw in S3 with the source payload preserved, so a bad load can be replayed rather than reconstructed.
- Promote raw files into standardised, queryable tables through a Medallion architecture: schema enforcement and type coercion on the way in, deduplication and conformed keys on the way up, incremental loads with full error handling throughout.
- Design and run scalable AWS pipelines (Glue, Lambda, S3, Redshift, Athena) feeding analytics and AI workloads, orchestrated with Airflow.
- Model the standardised layer with dbt on Databricks, and enforce data quality as a first-class product: test frameworks covering not-null, uniqueness, referential integrity and business rules, plus schema standards and lineage documentation.
- Govern the platform through Databricks Unity Catalog — one catalog for access control, lineage and discovery across every layer, so who can read what is a property of the data rather than a property of the cluster.
- Saw what the team's fragmented transformation logic was costing us, evaluated the options, and led the migration to dbt. Now maintain the standards the rest of the team builds against.
- Ship through Harness for dev, SQA and production infrastructure, with Bamboo driving on-prem ETL jobs. Own monthly release cycles end to end, including the change and task records that go with them.