Job Specification: Senior Data Engineer (Databricks & ADF)
Role Overview
We are seeking an experienced Senior Data Engineer to build, optimise, and operationalise data solutions on our Azure Data Lakehouse using Databricks, ADF, and Delta Lake. This is a senior individual contributor role focused on hands-on engineering, delivering high-quality data pipelines, and supporting the organisation’s Data, AI, and BI roadmap.
Key Responsibilities
Data Engineering & Pipelines
- Develop, code, deploy, and maintain Databricks notebooks, jobs, and pipelines as a core part of the role.
- Build and optimise scalable ETL/ELT workflows using Databricks, ADF, and Delta Lake.
- Ensure high-quality, reliable, and reusable datasets and data models.
Data Platform Delivery
- Work hands-on across ADLS, ADF, Databricks, and related Azure services to support end-to-end data processing.
- Implement automated testing, version control, orchestration, and CI/CD deployment for Databricks and pipelines.
- Collaborate with analytics and BI teams to deliver trusted data for insights and reporting.
AI & Data Science Support
- Enable model development, feature engineering, and ML workflows running on Databricks.
- Support deployment and scaling of AI/ML workloads across the platform.
Governance, Security & Quality
- Apply strong data governance practices including data quality, lineage, metadata, and access control.
- Ensure compliance with security and performance standards across the cloud data environment.
Collaboration & Leadership
- Provide technical leadership on complex data engineering topics.
- Support the evolution of the organisation’s Data, AI, and Business Intelligence roadmap.
- Act as a mentor and subject matter expert for modern data engineering practices.
Required Skills
- Essential: Strong hands-on experience coding, deploying, and managing Databricks notebooks, jobs, and pipelines.
- Proficiency in PySpark, SQL, Delta Lake, and modern data engineering techniques.
- Practical experience with ADF, ADLS, orchestration, and CI/CD in Azure environments.
- Experience supporting analytics and AI/ML workloads in production.
- Knowledge of data governance, security, and performance optimisation.
- Strong communication and cross-functional collaboration skills.
For more information please call Michael on 01 614 6058 or e:[email protected]
