Spark, query optimization, query execution acceleration, data lake house, hudi contributor, stream processing

distributed system, Java, C/C++

# Highlights:

Developed background table redistribution which lifted the day-1 limit of resize in redshift. The feature endures 4 million tables with 21 PB data in the first 6 months of release. Also owned feature testing, operations and fleet monitoring.

Independently delivered redistribution optimizations, resulting in throughput increases by 25%~100%. Brought down resize of 1 PB cluster to 3 days.

Implemented a reusable background job framework offering automated lifecycle management and parallel execution. It supports 15+ use cases cross-team. As a result of my effort onboarding the auto table optimization job, we saw a 120% increase in execution cycles fleet-wide, which greatly boosted the rate at which customers benefit from recommended table properties.

Led and mentored multiple projects including cost estimation of DDL queries, user interface on fast encryption progress, background table skew remediation.

Developed a framework which improves the visibility and availability of arbitrary database runtime states by externalizing them to a centralized data store. Auto table optimization as one of the consumers is able to bump its coverage from 2% to 92% for serverless Redshift, where databases auto pause frequently.

Aborting Training, Amazon Web Services, Inc.

SDE Intern, AI Vertical Services(7961) Team

• Solved customer’s pain points by delivering an API which allows the abortion of a solution version creation, a functionality that is absent from the AWS Personalized service

• Implemented the backend logic of the aborting training with Simple Workflow, which is used to build up the workflow that manages cloud resources

• Add console changes for the frontend

• Technologies: Mockito, PowerMocks, AWS Simple Workflow, Simple Workflow, Docker, Java