AWS Data Engineer

Roles & Responsibilities

Design and architect AWS Data Lake/Lakehouse solutions across Landing, Transformed, and Curated/Consumption zones.

Define and govern data architecture standards, patterns, and best practices.

Design reusable ingestion pipelines for REST APIs, JDBC databases, S3, and SaaS connectors such as Salesforce via AWS AppFlow.

Define data storage strategies covering hot, warm, and cold storage, encryption, and lifecycle policies.

Develop and deploy pipelines using AWS Glue, Lambda, Step Functions, EventBridge, and API Gateway.

Build batch and streaming transformation workflows using AWS Glue and Amazon Redshift.

Implement pipeline orchestration, monitoring, logging, and notification frameworks.

Develop and maintain the AWS Glue Data Catalogue, including schema evolution and metadata tagging.

Implement data security and access controls using AWS Lake Formation, IAM, and Secrets Manager.

Ensure compliance with data classification, retention, and audit requirements.

Support data quality and observability frameworks.

Monitor platform health, performance, and pipeline reliability, and troubleshoot failures and data quality issues.

Maintain architecture documentation, pipeline configurations, and operational runbooks.

Optimise AWS platform performance and cost efficiency.

Requirements

Essential

3–5 years of experience in data engineering, data architecture, or cloud infrastructure.

Hands-on experience with Amazon S3, AWS Glue, Amazon Redshift, Lambda, Kinesis, Step Functions, EventBridge, AppFlow, and Lake Formation.

Strong SQL skills and proficiency in Python or Scala.

Experience designing and implementing Data Lake/Lakehouse architectures.

Strong understanding of data governance, data cataloguing, and metadata management.

Experience with batch and streaming data processing.

AWS Certified Data Engineer – Associate, AWS Certified Solutions Architect, or equivalent.

Preferred

Experience integrating Tableau or similar BI tools through Redshift or S3.

Familiarity with MLOps and AI/ML deployment using AWS SageMaker.

Salesforce integration experience using AWS AppFlow.

Knowledge of CDC and incremental data loading patterns.

Experience in a government or public sector data environment.