6 Month Contract - Excellent Day rate
Dublin based - Hybrid
Primary Responsibilities
· Developing ETL pipelines in and out of data warehouse using combination of Java/Scala/Python Spark jobs for data transformation and aggregation
· Writing SQL queries against Snowflake.
· Provide production support for Data Warehouse issues such data load problems, transformation translation problems
· Develop unit tests for transformations and aggregations
· Develop production grade real time or batch data integrations between systems
· Real time data processing of events from Kafka using streaming processing into data warehouse.
· Design and build data pipelines of medium to high complexity
· Translate requirements for BI and Reporting to Database design and reporting design
· Understanding data transformation and translation requirements and which tools to leverage to get the job done
· Design and build machine learning pipelines of medium to high complexity
· Execute practices such as continuous integration and test-driven development to enable the rapid delivery of working code.
· Deploy production grade data pipelines, data infrastructure and data artifacts as code.
· Develop estimates for data driven solutions
· Communicate technical, product and project information to stakeholders
· Establish standards of good practice such as coding standards and data governance
· Peer review code developed by others
Knowledge & Skills
· Minimum BSc or BTech / B.E in Computer Science, Engineering, or related discipline.
· Relevant professional qualification such as AWS Certified Big Data, SnowPro Core certification, other Data Engineer certifications
- Strong development hands-on background in creating Snow pipe, and complex data transformations and manipulations using Snow Pipe, Snow SQL
- Hands-on experience with Snowflake external tables concepts, Staging, Snow scheduler & performance tuning.
· Good understanding of Snowflake Time travel concepts and zero-copy cloning, Network policies, clustering, and tasks
· 5+ year experience working in an enterprise big data environment
· Deep knowledge of Spark, Kafka and data warehouse such as snowflake, Hive, Redshift etc
· Hands-on experience in development, deployment and operation of data technologies and platforms such as:
o Integration using APIs, micro-services and ETL patterns
o Low latency/Streaming, batch and micro batch processing
o Data platforms such as Hadoop, Hive, Redshift or Snowflake
o Cloud Services such as AWS
o Cloud query services such as Athena
o DevOps Platforms such as Gitlab
o Containerisation technologies such as Docker and Kubernetes
o Orchestration solutions such as Airflow
· Deep knowledge of key non-functional requirements such as availability, scalability, operability, and maintainability
· Deep knowledge of SQL
· OS knowledge particularly Linux
· High on social skills, team spirit and empathy.
· Willingness to take ownership and ability to show strong personal commitment for the department and the team goals; must be comfortable with being considered a reliable and proactive influential team member who is not afraid to take on responsibility in the team.
· Ability to communicate clearly to business analysts and stakeholders as well as technical representatives.
· Very strong and proven communication and coordination skills, open-minded and determined.
· Responsible for planning, highlighting, and implementing possible improvements for existing and new applications.
· Good to have:
· Migration experience to Snowflake
· Hands on experience with Oracle RDBMS
· Exposure to Streamsets, DBT or other ETL tool
For more info and a confidential discussion please call Michael on 01 6146058
