UK Job Market Data Pipeline
ABOUT THIS PROJECT
An end-to-end Azure data engineering pipeline that ingests, transforms, and visualises live UK job market data from three sources — Reed API, Adzuna API, and a Kaggle CSV — fully automated via GitHub Actions.
Architected a star schema in ADLS Gen2, transformed data with Databricks Serverless (pandas over PySpark — the right call for 648 rows), queried via Synapse Analytics external tables, and visualised in a 5-page Power BI dashboard. Hit real Azure policy restrictions along the way: VM quota blocks, Azure Functions deprecation, and tenant-level storage denials — and solved each one with a defensible alternative. Total cloud spend: £1.81.
WHAT I LEARNED
Azure's free trial has hidden VM quota restrictions — Databricks clusters silently fail to provision, and the fix isn't in the docs, it's in the subscription settings
Databricks Serverless starts instantly but blocks standard PySpark auth methods — reading ADLS Gen2 via HTTP requests with a SAS token is a perfectly valid workaround for small datasets
GitHub Actions is a better scheduled ingestion trigger than Azure Functions for student subscriptions — free, reliable, and the 40-second runtime proved it
Synapse Analytics external tables over ADLS Gen2 work cleanly once you use credential-based data sources — the BLOB_STORAGE shortcut isn't always supported
The honest story of a project (quota errors, policy blocks, £1.81 in cloud spend) is more impressive to engineers than a clean run — real pipelines always involve workarounds
TECH STACK