The backpressure pattern consists of a feedback mechanism that allows consumers to inform upstream components when they are ready to handle new messages, preventing them from becoming overwhelmed or stressed. For feature stores, this means that there must be a mechanism to prevent feature pipelines from overloading the feature stores when writing features. In particular, updates by feature pipelines to online feature stores should not affect the performance of clients reading from the same online feature store.
It is important to consider how backpressure is handled in a feature store to ensure that the feature pipelines do not overload the system and cause performance issues. Proper handling of backpressure can help maintain the reliability and efficiency of the feature store and prevent missed SLAs, downtime, or data loss.
In Hopsworks, Kafka implements backpressure for feature pipelines, as it acts as an infinite buffer, decoupling the feature pipeline from the online feature store. A Hopsworks connector synchronizes feature updates from Kafka to the online feature store (RonDB) at high throughput and low latency.
.svg)