Developed batch big data pipelines in Scala and Python using Scio, Luigi, and Flyte running on GCP Dataflow.

Created datasets that were used for surfacing insights to creators on Spotify for Artists.

Collaborated with machine learning engineers on the implementation of pipelines to produce feature data for model training and batch inference.

Combined system telemetry with product data to derive metrics for monitoring the performance of forecasting models.

Collected and centralized event data output from multiple microservices to gain a deeper understanding of system states and their impact on the health of the product.

Produced data models and built data warehouse objects using Athena on top of an S3 data lake to enable internal users and services the ability to easily query change data that was captured from transaction logs on multiple database systems.

Responsible for maintaining and optimizing ever-growing continuous data pipelines comprised of microservices and Apache Flink applications along with the deployment, configuration, and scaling of the Apache Flink engine on a Kubernetes cluster.

Also dabbled in constructing and modifying cloud infrastructure and networking with Terraform.

Designed and architected a streaming change data capture pipeline to transport data from several different cloud-based database systems residing in multiple regions to a centralized data lake.

Developed ELT processes to load data into a Snowflake Data Warehouse then transform the data in micro-batches using stored procedures to facilitate analysis by internal customers.

Integrated Apache Flink pipelines, Apache Airflow workflows, and custom implementations with managed services such as Kinesis and S3 in AWS and BigQuery in GCP.

Played a major role in defining the standard for semi-structured diagnostics logging and telemetry analysis.

Contributed to the development of microservices and shared libraries in C# that were deployed in Microsoft Azure.

Collaborated with team members on many phases of the SDLC including but not limited to the production and review of technical specifications, the design and development of features and performance enhancements, the creation and execution of unit and integration tests, and the troubleshooting and resolution of defects.

Utilized software engineering techniques such as Object-Relational Mapping, Feature Flagging, Dependency Injection, and Caching.

Formulated data models then developed the corresponding database design and batch ETL processes needed to centralize data from both on-premise and hosted systems to a data warehouse.

Produced SQL statements of varying complexity to respond to data requests and perform data verifications.