Sensei: Gen-AI Platform (NYC)

Led a team of 10 engineers to build Sensei, a Generative AI feature for AWS Glue. Sensei enables users to create, manage, and troubleshoot Data Integration (DI) jobs via natural language, leveraging LLMs for script generation and recommendations. Key features include a Jupyter Lab plugin (Typescript/React), API layer (Spring Boot, AWS Bedrock), RAG APIs, and AWS Lambda orchestration for prompt persistence and feedback. Designed multiturn conversation support for contextual understanding.

Tech: Python, Spring Boot, AWS Bedrock, OpenSearch, EC2, S3, Docker.

AWS Glue 5 Architecture Upgrades (NYC)

Drove major architectural changes for Glue 5, optimizing artifact distribution by downloading all user artifacts on the driver. Developed a custom Spark file replicator (20% faster than default), Python module installer with virtual environment creation, multithreaded installation, proxy interceptor, and efficient file replication for large-scale deployments.

Tech: Java, Scala, Python, EC2, S3, Docker, JMeter, Netty NIO.

AWS Glue Connectors (NYC)

Led development of AWS Glue ETL Platform, managing 8 engineers and 10 contractors. Built frameworks for connector detection/certification, connector release to region-specific S3s, and support for data lake formats (Hudi, Iceberg, Delta). Enabled governed tables for cross-account use and integrated Baikal Redshift connector for native Glue job support.

Tech: Java, Scala, Python, EC2, S3, Docker.

Impact:

Delivered scalable, high-performance solutions for AWS Glue, leveraging Gen-AI, distributed systems, and modern data engineering frameworks.

Responsibilities:Supervised and Mentor a team of 7 software engineers that are responsible on working on below 3 layers

• Service layer (BAS – Bloomberg Application server): These Services are build using C++ tech stack, running on premises machines. I have built the services that is responsible to talk to another set of Services to fetch index calculation data. the response from other services is in chunks if the payload is more than a certain threshold. We have designed these API’s so that all response chunks will be processed concurrently using C++ futures. It optimizes the performance considerably.

• UI Layer (Type script): I have built the grid to display the Index calculation results for particular dates. It uses PUSH technology similar to WebSocket’s API, but We have used Bloomberg inhouse mechanism called DMP (Data Model Publishing) to implement that. All UI components are built using types script.

• Index Publishing Framework (Python 3): These Framework APIs are responsible to publish the index data in bulk (as a batch) or on demand using API call. These internally using Bloomberg third party API’s for calculating the index data for the given period. Then using Python Pandas API’ (Data frames) s to massage the data received and push it on HDFS in the form of Parquet files. That finally got persisted into Cassandra.

Responsibilities: Supervised/ Mentor a team of 10 engineers’ team in which 7 are software engineers and 3 are Data Ops engineers that are responsible in 3 streams of work:

• Design and develop Generic Microservice, that includes: Single API that supports all methods (GET, POST, PUT, DELETE). This API is providing capability of Posting and retrieving the Orders/Security/Trade data back and forth from Alfa software to Aladdin platform. It works in both mode Sync and Async.

• Design the Posting Server in generic way: It includes

o Build Order Positing server and Security Posting server. All these servers are subscribing to Kafka Topic and listening to Order and Security data. Then transform this data and then pushed it in Alfa.

o Build the Generic API Metadata Model that allows the configuration driven way, to determine for which message (Order /Security) which set of Alfa API needs to be invoked. So, it’s very scalable and adhere to Open-Close design paradigm.

• Design and develop Infra as a Service Strategy to Build and deploy Alfa product in Blackrock Eco system: It includes

o Designing the dev ops model for building the pipeline using terraform to create AMI (Amazon Machine Images) for the given microservices.

o Design GitLab architecture to build the artifacts for Java based microservices.

o Now Working on designing the server less way of building the docker images, push them it into Helm Repo and then deploy on Kubernetes cluster.

Project Details:

Building the Asset Management Analytics Platform, that provides Data scientist the ability to create / deploy Business Model as Data sets in the platform. Also provides ways to source data (ingest Data) from different sources into the platform. Platform also provides the capability to expose these data sets as DaaS (Data as a Service) API’s. Platform also provides the UI to view these Datasets as Data Catalogue.

Technologies used- Java 8, Apache Spark, Spring Boot, Livy, Jupyter Notebook, Maven, Data Stax Cassandra, React.

Responsibilities:

• Design and develop the Generic Data as a Service (DaaS) API: Involve in designing and developing the microservice based API’s similar to Quandle API’s (https://docs.quandl.com ) , that provides a way to request particular data set with request parameters and response back the data sets in the form of table or timeseries.

• Design and develop the Reusable components for the platform: I have design and developed reusable components like:

o Since Data Stax version of Cassandra do not have any ORM like API’s, so built the light weight custom Data Access Layer that works like Mybatis API (CQL to Object Mapping).It can run simple CQL or SOLR based queries with same interface.

o Built the Generic Access Control Service API to provide a centralized way of accessing entitlements.

o Built the Generic Excel Upload/Download API’s.

• Design and develop the Configuration based Light Weight Data ingestion Framework using Spark - It provide a way to source data from files ( Parquet , Avro , CSV ) , RDBMS , NOSQL by providing the configuration settings .I am working on making it like Plugin driven , so that if custom Transformation /filtration is required , that can be provided as an external dependency.

• Provide an integration /configuration strategy to use Grafana (https://grafana.com/ ) into the platform API for visualization purpose.