Lead Data Engineer

About this role

The Data Products Team builds and certifies the data products that power outage management and reliability reporting. The team owns the full data lifecycle, from raw source feeds to validated, stakeholder-ready outputs. We’re looking for a hands-on Data Engineer to build and maintain ETL pipelines, implement data quality validations, and support the delivery of certified, versioned data products. You will work closely with the team and operational stakeholders to ensure data accuracy, reliability, and readiness for downstream reporting and business use.

Responsibilities Build and maintain scalable ETL pipelines using AWS Glue, Spark, and Athena to process modem telemetry, outage events, and customer impact data. Develop and manage daily data quality checks, including SQL validations, threshold-based alerts, and row-count monitoring. Investigate data anomalies such as nulls, duplicates, missing intervals, and unexpected data distributions, resolving issues before they affect downstream systems.

Integrate data from multiple upstream systems, including AOM, IPDR, Scope, CARS, IRIS, and Gisual , to produce unified, validated datasets. Support data product certification by validating outputs and ensuring they meet established quality standards before stakeholder delivery. Coordinate versioned data product releases, including test case development, release notes, go/no-go decisions with end users, and patches through a structured release process.

Document data validation methodologies, pipeline behavior, and release processes. Collaborate with data engineering teammates and BI and operational stakeholders throughout the data delivery lifecycle. Qualifications Strong proficiency in SQL , including window functions (LAG, PARTITION BY), grouping sets, null handling, and Athena-specific query patterns. Experience with Python for scripting, automation, and data validation.

Hands-on experience with AWS data services, particularly Glue, Athena, S3, EMR, and Step Functions . Experience using Spark for large-scale data processing, preferably on EMR. Strong understanding of data quality and the ability to investigate anomalies, identify root causes, and determine when data does not look correct. Experience validating data pipelines and ensuring data accuracy and consistency before downstream consumption.