Willow is an open-source framework for building Change Data Capture pipelines. It reflects database changes to a cache in near real-time. Read more at https://willow-cdc.github.io/
• Developed framework to simplify Change Data Capture pipeline creation from 9 steps to 4 steps, leveraging open-source tools such as Debezium and Apache Kafka.
• Resolved Debezium configuration issues which prevented multiple pipelines from connecting to the same database (PostgreSQL, Debezium).
• Ensured Willow works properly when provided a minimum privileged database user, avoiding the need to provide Willow with superuser privileges.
• Architected and implemented a RESTful API, facilitating pipeline creation and teardown (Node.js, Express).
• Designed and constructed a custom Kafka connector to transform Kafka events and send to a cache (Node.js, KafkaJS, Redis).
• Built user-friendly UI that streamlines pipeline management (React, Material UI).
• Authored technical case study, providing an in-depth analysis of Willow's problem domain, system design, and engineering decisions (willow-cdc.github.io/case-study ).