Hopsworks 3.4 Product Release

Hopsworks 3.4 is now generally available. This version adds support for multi-region high availability and integration with external Kafka clusters. With 3.4, we now allow users to customize the operating system libraries in their Python environment. We have also added new support for scheduling Python and Spark jobs (e.g., used to orchestrate machine learning pipelines).

Multi-Region / High Availability

With the 3.4 release, Hopsworks introduces support for high availability across different geographical regions. The Hopsworks online feature store supports active-active replication, while the offline feature store supports active-standby operation. This enables Hopsworks to continue operation, for both batch and online feature store operations, even if an entire region goes offline by falling back to the other region without any loss of service.

Reach out to the Hopsworks team to learn more about this capability.

Integration with external Kafka

Hopsworks supports Apache Kafka for the ingestion of data to the feature store. By default, Hopsworks comes with an embedded Kafka cluster managed by Hopsworks itself. Starting from version 3.4, users have the option to configure Hopsworks to leverage an existing external Kafka cluster in the cloud or on-premises.

Learn more about the bring your own Kafka functionality

Job Scheduling

Hopsworks 3.4 introduces a new improved way of scheduling machine learning pipelines. We developed an improved UI, and job scheduling has become significantly simpler and easier to use with the 3.4 release without having to rely on an external orchestrator, such as Apache Airflow. The new job scheduling functionality is also supported by the Hopsworks Python library, allowing users to manage the job scheduling from their CI/CD pipelines.

Learn more about the new job scheduling functionality.

Custom Docker Commands

Hopsworks 3.4 improves the customization options for the Python environments powering Python/PySpark jobs, Jupyter notebooks, and model serving on Hopsworks. Users can now execute custom shell commands to install operating system libraries or to download additional non-Python libraries. We also provide UI access to the history of the project Python environment, where you can see which library has been installed, updated or deleted at every stage.

Learn more about about the custom docker command and the Python environment history

Online Feature Store Client Improvements

In the 3.4 release, Hopsworks has extended the REST API for the online feature store, released in Hopsworks 3.3, by providing improved support for complex data structures including metadata with associated logging, metrics and alerting functionality.