
Goal
Modernise and centralise data management for HM Courts & Tribunals Service (HMCTS), part of the UK Ministry of Justice. Support the migration of legacy data workloads to a cloud-based Strategic Data Platform (SDP), improving data accessibility, reliability and operational efficiency.
Role
- Delivered cloud-based data migration and transformation: Developed and enhanced SQL and PySpark workflows in Azure Synapse Analytics, supporting the migration, transformation and consolidation of data into the HMCTS Strategic Data Platform.
- Automated data processing and deployment: Developed CI/CD integration scripts supporting data loading, transformation, validation and writing to the target platform, improving the repeatability and maintainability of data engineering workflows.
- Improved data accessibility and reduced maintenance costs: Contributed to the cloud migration initiative, achieving a 25% reduction in data access time and saving thousands of pounds in annual maintenance costs.
Capability
Cloud-based data engineering and modernisation enabled through:
- Data Accessibility: Made consolidated datasets more accessible for downstream analysis and reporting.
- Data Migration: Migrated and consolidated data from existing systems into a centralised cloud platform.
- Data Transformation: Developed SQL and PySpark notebooks in Azure Synapse to process, validate and prepare datasets for downstream analytics.
- Data Integration: Supported data ingestion and movement through Azure-based data services.
- Automation and Reliability: Used CI/CD practices to support repeatable data loading, validation and deployment.
Stack
- Python / PySpark – Data processing and transformations
- SQL – Data querying and transformation
- Microsoft Azure
- Azure Data Factory (ADF) – End-to-end data pipeline orchestration and integration.
- Azure Databricks – Upstream data cleansing and PySpark-based data prep.
- Azure Synapse Analytics – Downstream SQL and PySpark transformations and analytics.
- CI/CD – Automated integration and deployment workflows
- Git – Source control
- Regular Expressions (Regex) – Text processing and data manipulation
Year
2023