Data Engineer - Python & AWS (Public Sector)

1-year contract, renewable

Hybrid work arrangement

Government project

Role Overview

The Data Engineer will join the Data Engineering & Infrastructure team to support the operations, maintenance, and enhancement of ODIN, working under the direction of the Team Lead and alongside PSD officers, appointed vendors, source system owners, and business users. The role provides hands-on data engineering capacity for ODIN's ingestion, transformation, data quality, and downstream data services.

This is a hands-on, delivery-focused role. Responsibilities include developing and maintaining ETL pipelines and processing logic, working with existing ODIN data models and datasets, investigating legacy and inconsistent HR data, automating validation and operational processes, executing testing, maintaining technical documentation, and enabling reliable downstream consumption of HR data. The role operates within established PSD processes, security requirements, technical standards, and delivery priorities; key decisions, stakeholder commitments, and approvals remain with the relevant PSD officers and system owners.

Key Responsibilities

Data Pipeline & Interface Engineering — Develop and maintain ETL pipelines and file-based interfaces for ingesting HR data into ODIN, including data profiling, mapping, specifications, and dataset onboarding

Data Transformation & Modelling — Build and maintain SQL and Python logic to cleanse, standardise, transform, and reconcile HR data into reliable, reusable datasets, including historical and legacy data

Data Model & Dataset Maintenance — Enhance ODIN data models, tables, and dependencies to meet new requirements, with traceable mappings, definitions, and processing rules

System Enhancements & Testing — Support CRs and SRs by reviewing requirements, executing tests, validating outputs, documenting defects, and resolving issues through implementation

Data Quality, Validation & Lineage — Automate validation, reconciliation, and cleansing checks; investigate anomalies, trace lineage, resolve root causes, and verify fixes

Pipeline Operations & Reliability — Monitor pipelines and interfaces, troubleshoot failures, perform reruns, and improve reliability, performance, maintainability, and operational documentation

Downstream Data Consumption — Develop governed datasets, Athena tables/views and queries, and Tableau data sources/workbooks for approved reporting and analytics needs

Technical Collaboration & Delivery — Translate stakeholder requirements into data mappings, transformation rules, and technical solutions, while documenting decisions and supporting timely delivery

Audit & Governance Support — Provide technical evidence for audits, access reviews, and assurance activities, and support required remediation

Requirements

Technical Skills — Strong proficiency in SQL and working proficiency in Python, with hands-on experience developing data pipelines, transformations, validation, and automation. Experience with AWS data services such as S3 ingestion, CSV/Parquet, IAM, partitioning, and Athena tables, views, and queries (or comparable cloud data platforms) is preferred. Familiarity with Spark, Databricks, or similar modern data engineering technologies is advantageous but not required for the current ODIN environment

Data Engineering & Data Warehousing — Hands-on understanding of ETL, data pipeline design and operations, relational and file-based data, data warehousing, and data modelling. Able to work with complex or legacy source structures and transform them into clean, standardised, and reliable datasets for downstream consumption

Data Quality, Governance & Troubleshooting — Strong analytical and problem-solving skills, with practical experience implementing data validation and reconciliation checks, investigating discrepancies, tracing data lineage, and identifying root causes across source, interface, and transformation layers

Thinking Clearly & Making Sound Judgement — Ability to examine complex problems, assess dependencies and risks, and propose workable solutions or escalation paths with clear supporting analysis

Working Effectively with Stakeholders — Strong communication and collaboration skills, with the ability to work effectively with business users, technical teams, source system owners, and vendors, and to explain technical findings clearly

HR Data Management & Visualisation — Keen interest in HR data or workforce processes. Experience with Tableau or similar BI tools, including development or maintenance of datasets and workbooks, is advantageous

Delivery & Collaboration — Able to work independently, manage priorities, and deliver quality engineering work on time. Maintains clear documentation, follows change, UAT, and incident processes, translates requirements into practical solutions, collaborates effectively with stakeholders, and escalates risks promptly

/Ref: T24-239