Data lake team member creating central repository for all data throughout the company. Created Kafka producers across all touchpoints in codebase to centralize all data into a single location, as well as, Kafka consumers to stream data into HDFS. Established nightly ETL using oozie workflows and hive/spark to process data from HDFS into hive managed tables for consumption on the front end through a presto cluster.

Created libraries for creating Kafka Producers and Consumers and Streaming data out to HDFS. Leveraged .NET TPL to create highly efficient and threaded code used to process hundreds of millions of rows daily into Hadoop.