Job Description

Main duties & responsibilities

You will be working on a new data platform, within a large digital transformation programme that incorporates data warehousing, cdps and ciams.  You will be working with a team of Engineers, Testers, Delivery Managers and Product Owners to develop a data platform that will be central to the wider solution. Hosted within Azure technology, using some of the latest tools available.

You will be required to:

• Design, build and unit test aspects of data services
• Input into high level design and responsibility for low level designs
• Hands on development of data pipelines, transformation and modelling within Microsoft Fabric
• Owning your features from build through to production deployment.
• Creation of accurate, insightful and informative technical documentation
• Build data infrastructure using Bicep templates, deployed via Azure DevOps

The ideal candidate will be:

• A talent for design with a keen eye for detail, structured thinking and best practice for performant data flow.
• A self-starter who can quickly develop an understanding of a complex platform.
• An excellent communicator who is confident having technical discussions with senior stakeholders.

Must have skills:

• Spark/Pyspark, used for data transformation and modelling. Ideally across a multi-layer data lake.
• Python
• Experience in API integration (via use of Function Apps, Logic apps or a similar tool)
• Use of data pipeline tools (Azure Fabric/Data Factory/Airflow/a similar tool)
• Ability to create infra as Code within Azure (ARM or Bicep).
• SQL, ideally Microsoft SQL
• Strong understanding of Azure cloud architecture
• Experience within automated testing

Would like to have skills:

• Experience using Microsoft Landing Zones architecture
• Microsoft Fabric
• Delta Lake (or a similar Lakehouse technology)
• Network security with Azure
• Streaming (Fabric EventStreams, Steam Analytics, Event Hubs, or another tool used for this purpose)