Implemented realtime system usage event capture in the production stack across multiple microservices
Singlehandedly designed and implemented the company’s first iteration of an analytics data processing pipeline in 2019
• Batch ingestion, batch transformation, storage, presentation via BI tools (AWS Quicksights)
• Built with AWS s3, AWS lambda functions, AWS SQS, AWS kinesis firehose, redshift, dbt.
Designed and implemented second iteration of data processing pipeline in 2021 with primary goal of scalability.
• Implemented generalized producer consumer framework that eliminated bottlenecks for scaling the previous pipeline design by streaming data instead of batch processing.
• Reduced latency from event capture to end reporting from 1 hour to < 1 minute.
• Ingested ~1MB/min during peak loads.
• Built with the custom python pipeline framework, AWS SQS, kibana, redshift, dbt, and elasticsearch.