Build parts of RAG at Meta SuperIntelligence Lab.

• > Design and build backend On-demand framework around big data processing system and developed collection move/rename orchestration, which supports the bulk move/delete process and onboarded Microsoft Customers.

• > Redesign the architecture of Path pattern rule update that reduces the job runs and saves the customer costs.

• > Optimize the spark jobs using spark graphframes to increase the throughput that helps support more customers.

• > Engineering Excellence:

• Built a part of monitoring and metrics system for each orchestration which helps identifying the live sites issues and resulted in the drop of higher sev incidents, saving the internal costs.

• Enabled Batch processing platform for Sovereign clouds, which on-boarded 300+ customers.

• Design and build the architecture of Voice Identification for a customer support and develop ML pipeline using Spark, Kinesis and Pytorch

• Build ETL pipeline for Product Prediction using Clickstream on Kafka, Timeseries DB, Redis, Spark and MySQL.

• Deploy applications on AWS (Lambada, Kinesis, RDS, EMR, Sagemaker) docker and Kubernetes based Argo AI.

Design and development of backend microservices to provide ML and other data science tool as a service

Worked on training, prediction and Evaluations of ML models using various ML library frameworks like Scikitlearn, XGboost, Keras on Tensorflow and Spark, PMML

Built libraries on Spark and Python; wrote templates to run on distributed spark running on HDFS and Hive

Deployed Web applications in Container-based system using docker and kubernetes.

Tech Stack: Python, Scala, R, docker, kubernetes, cloudant, Java, redis, flask, Maven