Big Data Engineering

Batch Processing Platform Development on AWS

• Implemented a batch processing platform on AWS using Spark and MySQL, by combining 70 GB New York yellow taxi records and weather records, to predict passenger hotspot in rainy day with DBSCAN algorithms. (AWS, Spark, MySQL)

• Decreased running time by 25 times through parallel computing on Spark with 5 worker nodes.

• Visualized predicted hotspots and statistical results on website by Flask. (Flask)

Project 1: CF-Lung research database

• Designed an Enhanced Entity Relational(EER) model and implemented it. (PostgreSQL, Python)

• Built a novel CF-lung research database which combining both EHR and experimental data (eg. Genomics data).

Project 2: Worm tracking through image analysis

• Automatically ingested time serials worm image data, processed image by using segmentation algorithm and generating numerical data by Julia.

• Improved precision of worm dead time point, and quantified worm features through images.