o Work in the data platform team, and help build a data development platform which schedules and monitors around 2 million ETL jobs daily using Springboot, Thrift, ElasticSearch, Zookeeper, Kafka, Flink, SQL.
o Designed and implemented new data conversion features to support growing business requirements which made creating new ETL jobs 10x faster for new users and greatly reduced the latency and resource usage of the pipelines.
o Participated in system performance improvement project; Designed and integrated offline data pipeline with hive and clickhouse to reduce api latency and DB load 100X; Optimized scheduling algorithm to reduce 60% system latency;Implemented read/write separation for web services to support better response time.
o Drove the initiatives of authorization migration from user to account level;Discussed with multiple teams to collect feedback, and defined the requirements of the service; Designed and implemented the migration component in our services which supported the smooth migration of 3+ business lines, and 1000+ tasks.