Job Specification: Senior Data Engineer (Databricks & ADF)

Role Overview

We are seeking an experienced Senior Data Engineer to build, optimise, and operationalise data solutions on our Azure Data Lakehouse using Databricks, ADF, and Delta Lake. This is a senior individual contributor role focused on hands-on engineering, delivering high-quality data pipelines, and supporting the organisation’s Data, AI, and BI roadmap.

Key Responsibilities

Data Engineering & Pipelines

• Develop, code, deploy, and maintain Databricks notebooks, jobs, and pipelines as a core part of the role.
• Build and optimise scalable ETL/ELT workflows using Databricks, ADF, and Delta Lake.
• Ensure high-quality, reliable, and reusable datasets and data models.

Data Platform Delivery

• Work hands-on across ADLS, ADF, Databricks, and related Azure services to support end-to-end data processing.
• Implement automated testing, version control, orchestration, and CI/CD deployment for Databricks and pipelines.
• Collaborate with analytics and BI teams to deliver trusted data for insights and reporting.

AI & Data Science Support

• Enable model development, feature engineering, and ML workflows running on Databricks.
• Support deployment and scaling of AI/ML workloads across the platform.

Governance, Security & Quality

• Apply strong data governance practices including data quality, lineage, metadata, and access control.
• Ensure compliance with security and performance standards across the cloud data environment.

Collaboration & Leadership

• Provide technical leadership on complex data engineering topics.
• Support the evolution of the organisation’s Data, AI, and Business Intelligence roadmap.
• Act as a mentor and subject matter expert for modern data engineering practices.

Required Skills

• Essential: Strong hands-on experience coding, deploying, and managing Databricks notebooks, jobs, and pipelines.
• Proficiency in PySpark, SQL, Delta Lake, and modern data engineering techniques.
• Practical experience with ADF, ADLS, orchestration, and CI/CD in Azure environments.
• Experience supporting analytics and AI/ML workloads in production.
• Knowledge of data governance, security, and performance optimisation.
• Strong communication and cross-functional collaboration skills.

For more information please call Michael on 01 614 6058 or e:[email protected]