Greater Toronto Area, Canada
• Helped hire and establish a remote team in Canada. The incentive was to look for talent outside the US to build a globally diverse team. Helped with documentation and training videos to facilitate onboarding and mentoring of junior engineers that were hired in Canada. Grew the team's Canadian footprint from 1 to 5 engineers including myself.
• Additional resources in Canada allowed us to expand the scope of the current team to take charge of the platform data compliance needs due to GDPR, FedRamp, PCI, HIPAA, and customized data retention. The nightly pipelines managed by the team leveraged the lambda architecture and processed hundreds of terabytes of snappy compressed Avro serialized data.
• Manage a stateless service that is responsible for consuming data from Kafka and performing real-time ingestion in Dynamo, Elasticsearch, and Druid.
• Maintain a RESTFUL web service that exposes a rich query facade to query data from Elasticsearch, Athena, Druid, Dynamo, and RDS. This API is queried by customers and internal teams millions of times a day.
• Managed an Athena-powered datalake which is hydrated nightly by our EMR clusters.