CTM Intelligence

Software Engineer Intern on the FoundationDB Platform Team

• Designed and developed a real-time Performance Monitoring interface for over 180 million data points with Spark and Hive

• Reduced 20% of the false-positive performance anomalies via conducting repetitive analysis on the batch generated anomalies

• Built a thread-safe Hive query interface over JDBC in Python with Kerberos authentication support on Hadoop Edge Nodes

• Utilized Hive table partitioning to improve read query time when ingesting 3.6 billion data points from Spark into Hive

• Designed and implemented an Anomaly Detection system to identify abnormal signals in the financial option markets

• Eliminated 30% of false positives compared to the baseline model by building a configuration-driven Keras Deep Autoencoder

• Constructed a scalable machine learning pipeline processing over 2.5 million data points daily with Kafka and Cassandra

• Maintained reliability of the pipeline by achieving over 90% code coverage in automated Jenkins tests with Docker

• Migrated the security metadata storage system used by the Fixed-Income Data Platform team from HDFS to Cassandra

• Reduced Spark executor memory consumption by over 500% for online queries by integrating Cassandra with Spark

• Decreased online query latency by more than 20% by developing the Cassandra schema around the historical query patterns

• Created a data pipeline that ingests over 3 million historical and current price data points into the Cassandra cluster