Maps Data Processing

Build and maintain data pipeline software infrastructures, including batch data jobs scheduling system, real-time data streaming pipeline, monitoring/alerting, tooling, backup, data synchronization between clusters, etc.

Realtime user events streaming pipeline:

▪ Designed, developed, and owned the data streaming pipeline that handles user events and user attributes using Filebeat, Kafka, Flink on Kubernetes

▪ Handles data that averages at 600 event/s and peaks at 2000 events/s, with 95% of the events having latency below 2s

▪ Implemented and supported monitoring, alerting, and replay to ensure zero data loss

Batch data pipeline job scheduling system:

▪ Developed and owned the core batch data pipeline with customized Luigi framework, led the team for maintenance and support

▪ Built job templates to support different types of data ETL jobs (Sqoop, Hive, Spark, MapReduce, etc.)

▪ Designed and built auto synchronization between main data cluster and read-only data cluster (for ad-hoc query usage)

▪ Built data validation system to ensure data quality, monitoring system to track job delay, and CI/CD on Jenkins

• Carnegie Mellon University Student Studio Project Sponsored by RobotWits LLC.

• Implemented a self-driving simulation and automation testing system based on an open-source simulator.

• Designed the modular architecture to interact with Robot Operating System using a publish-subscribe pattern.

• Implemented a logging framework, which records the simulation session and supports its replaying.

• Managed project development cycle using agile approaches (Scrum) and project tracking tools (Jira).

• Software engineer intern, SDK development.

• Experience in distributed systems architecture design and programming

• Experience with No-SQL DBs (MongoDB, Redis)