We describe the capabilities that need to be added to Lakehouse to make it an AI Lakehouse that can support building and operating AI-enabled batch and real-time applications as well LLM applications.
Nearly a year ago, the Hopsworks team embarked on a journey to migrate its infrastructure to Kubernetes. In this article we describe three main pillars of our Kubernetes migration.
We present a unified software architecture for batch, real-time, and LLM AI systems that is based on a shared storage layer and a decomposition of machine learning pipelines.
Hopsworks has added support for approximate nearest neighbor (ANN) indexing and vector similarity search for vector embeddings stored in its feature store.
This article introduces a taxonomy for data transformations in AI applications that is fundamental for any AI system that wants to reuse feature data in more than one model.
In this article we introduce the snowflake schema data model for feature stores, and show how it helps you include more features to make better predictions
In this article, we cover the added value of a feature store over a data warehouse when managing offline data for AI.
Read how Hopsworks generates temporal queries from Python code, and how a native query engine built on Arrow can massively outperform JDBC/ODBC APIs.
Learn more about how Hopsworks (RonDB) outperforms AWS Sagemaker and GCP Vertex in latency for real-time AI databases, based on a peer-reviewed SIGMOD 2024 benchmark.
We present how Hopsworks leverages its time-travel capabilities for feature groups to support reproducible creation of training datasets using metadata.
In this article we explore how our Public Key Infrastructure has changed over the years coming to its current form, a Kubernetes first-class citizen.
In this tutorial, we walk you through how we install air-gapped environments in Hopsworks and how we extended this process to test air-gapped installations as well.
This blog explores how Hopsworks can support FSIs in addressing key regulations, including DORA, AMLA, PSD3, and more.
Discover the power of feature stores in modern machine learning systems and how they bridge the gap between model development and production.
In this article, we outline how we leveraged ArrowFlight with DuckDB to build a new service that massively improves the performance of Python clients reading from lakehouse data in the Feature Store
In this blog, we discuss the state-of-the-art in data management and machine learning pipelines (within the wider field of MLOps) and present the first open-source feature store, Hopsworks.
We explain a new framework for ML systems as three independent ML pipelines: feature pipelines, training pipelines, and inference pipelines, creating a unified MLOps architecture.
Explore the convergence of MLOps and DevOps. Learn about their purposes, differences, and areas of integration and discover best practices, challenges, and their future potential.
With support to Apache Hudi, the Hopsworks Feature Store offers lakehouse capabilities to improve automated feature pipelines and training pipelines (MLOps).
We review Python libraries, such as Pandas, Pandas2 and Polars, for Feature Engineering, evaluate their performance and explore how they power machine learning use cases.
Unlock the power of Apache Airflow in the context of feature engineering. We will delve into building a feature pipeline using Airflow, focusing on two tasks: feature binning and aggregations.
Find out how to use Flink to compute real-time features and make them available to online models within seconds using Hopsworks.
Time-series data consists of records in chronological order and is crucial for forecasting trends and patterns. In this blog, we take a look at how to use Facebook Prophet for time-series forecasting.
In this blog, we describe how we leverage the authentication and authorization support in Open Distro for Elasticsearch to make elasticsearch a project-based multi-tenant service in Hopsworks.
Deep learning is now the state-of-the-art technique for identifying financial transactions suspected of money laundering. It delivers a lower number of false positives and with higher accuracy.
Next up in our lineup is Tales Marra, Machine Learning Engineer at AssessFirst.
This week Rik met with Niels Naglé, Area Lead Data & AI at Info Support, and Joop Snijder, CTO at Aigency, for a quick chat.
Join Rik as he chats with David Colls, Founding Data Scientist at Nextdata, about Data & AI, the evolution of machine learning, and effective tech practices.
In this blog post, we have selected the 25 most-read dictionary entries from the MLOps and LLMOps Dictionary to highlight key trends and lessons in AI.
The third edition of the LLM Makerspace dived into an example of an LLM system for detecting check fraud.
In this post, we will look at how to put feature pipelines into production using Hopsworks.
We explore how to develop a compelling value case that quantify the benefits and costs associated with implementing a feature store.
This article covers the different aspects of Job Scheduling in Hopsworks including how simple jobs can be scheduled through the Hopsworks UI by non-technical users
A summary from our LLM Makerspace event where we built our own PDF Search Tool using RAG and fine-tuning in one platform. Follow along the journey to build a LLM application from scratch.
On the decision of building versus buying a feature store there are strategic and technical components to consider as it impacts both cost and technological debt.
This is a summary of our latest LLM Makerspace event where we pulled back the curtain on a exciting paradigm in AI – function calling with LLMs.
Redis will no longer be open source. Our own project, RonDB, will continue being open source in order to uphold the principles that keeps the technology advancing.
In this article we describe the software factory approach to building and maintaining AI systems.
Hopsworks has added support for Delta Lake to accelerate our mission to build the Python-Native Data for AI platform.
A tutorial of the Hopsworks Feature Query Service which efficiently queries and joins features from multiple platforms such as Snowflake, BigQuery and Hopsworks without data any duplication.
The rapid development pace in AI is the cause for a lot of misconceptions surrounding ML and MLOps. In this post we debunk a few common myths about MLOps, LLMs and machine learning in production.
At Hopsworks the F.A.I.R principles have been a cornerstone of our approach in designing a platform for managing machine learning data and infrastructure.
We go through the most common errors messages in Pandas and offer solutions to these errors as well as provide efficiency tips for Pandas code.
Following our previous blog, we expand on the architecture to fit a Tier 1 classification where all components of Hopsworks are replicated in a different geographical region.
A comparison of the online feature serving performance for Hopsworks and Feast feature stores, contrasting the approaches to building a feature store.
As a continuation to our last blog on the EU AI Act, this blog explores the implications of how Hopsworks machine learning systems can address the act's requirements for high risk AI applications.
Learn how Hopsworks achieves high availability for the Online and Offline Feature Store and a Metadata layer, making it an operational ML system both for on-premise and cloud infrastructure.
What is the corporate and societal significance of the EU AI Act and how does it impact organizations with high risk AI systems?
Data is evolving from traditional Business Intelligence to Artificial Intelligence and Machine Learning for predictive analytics, creating new requirements for how businesses operationalises.
Read about the advantages of using DBT for data warehouses and how it's positioned as a preferred solution for many data analytics and engineering teams.
This blog explores MLOps principles, with a focus on versioning, and provides a practical example using Hopsworks for both data and model versioning.
With the latest version of Hopsworks we introduce new capabilities such as running custom bash commands and a improved UI which shows you the history of the python environment.
Delve into the profound implications of machine learning embeddings, their diverse applications, and their crucial role in reshaping the way we interact with data.
A tutorial of how to use our latest Bring Your Own Kafka (BYOK) capability in Hopsworks. It allows you to connect your existing Kafka clusters to your Hopsworks cluster.
An ML model’s ability to learn and read data patterns largely depend on feature quality. With frameworks such as FeatureTools ML practitioners can automate the feature engineering process.
Explore the power of feature engineering for categorical features using Pandas. Learn essential techniques for handling categorical variables, and creating new features.
Hopsworks has successfully completed the AICPA Service Organization Control (SOC) 2 Type II audit.
Hopsworks has received an ISO 27001 certification, the globally recognized standard for establishing, implementing, maintaining, and continually improving an information security management system.
This blog analyses the cost-benefits of Feature Stores for Machine Learning and estimates your return on investment with our Feature Store ROI Calculator.
Read about Hopsworks 3.1 and the new improvements in the feature store (time-series splits for training data, support for managing thousands of models), stability and user-interface.
In this blog we present an end to end Git based workflow to test and deploy feature engineering, model training and inference pipelines.
Learn more about how Hopsworks stores both data and validation artifacts, enabling easy monitoring on the Feature Group UI page.
In this blog, we introduce Hopsworks Connector API that is used to mount a table in an external data source as an external feature group in Hopsworks.
Discover how you can easily make the journey from ML models to putting prediction services in production by choosing best-of-breed technologies.
Learn how the Hopsworks feature store APIs work and what it takes to go from a Pandas DataFrame to features used by models for both training and inference.
Hopsworks Serverless is the first serverless feature store for ML, allowing you to manage features and models seamlessly without worrying about scaling, configuration or management of servers.
Hopsworks is the first feature store to extend its support from the traditional Big Data platforms to the Pandas-sized data realm, where Python reigns supreme. A new Python API is also provided.
Hopsworks 3.0 is a new release focused on best-in-class Python support, Feature Views unifying Offline and Online read APIs to the Feature Store, Great Expectations support, KServe and a Model serving
In this blog post we showcase the results of a study that examined point-in-time join optimization using Apache Spark in Hopsworks.
An introduction to the EU AI Act and how Feature Stores provide a great solution to the obligations imposed by the regulation.
Programmers know data types, but what is a feature type to a programmer new to machine learning, given no mainstream programming language has native support for them?
Operational machine learning requires the offline and online testing of both features and models. In this article, we show you how to design, build, and run test for features.
We go through the new features and developments in Hopsworks 2.5 that will benefit open-source users and customers alike.
We introduce how to use the What-If Tool as a Jupyter plugin on Hopsworks to build better machine learning models by making it easier to ask counterfactual questions about your model’s behaviour.
Learn how to connect Hopsworks to Snowflake and create features and make them available both offline in Snowflake and online in Hopsworks.
We are introducing a new feature in Hopsworks UI - feature code preview - ability to view the notebook used to create a Feature Group or Training Dataset.
Learn how to set up customized alerts in Hopsworks for different events that are triggered as part of the ingestion pipeline.
In this blog post we demonstrate how to build such a pipeline with real-world data in order to develop an iceberg classification model.
Learn how to publish (write) and subscribe to (read) streams of events and how to interact with the schema registry and use Avro for data serialization.
Hopsworks brings support for scale-out AI with the ExtremeEarth project which focuses on the most concerning issues of food security and sea mapping.
Read about how the Hopsworks Feature Store abstracts away the complexity of a dual database system, unifying feature access for online and batch applications.
This tutorial gives an overview of how to work with Jupyter on the platform and train a state-of-the-art ML model using the fastai python library.
HopsFS is an open-source scaleout metadata file system, but its primary use case is not Exabyte storage, rather customizable consistent metadata.
Learn how to train a ML model in a distributed fashion without reformatting our code on Databricks with Maggy, open source tool available on Hopsworks.
This tutorial will show an overview of how to install and manage Python libraries in the platform.
Evolve your models from stateless AI to Total Recall AI with the help of a Feature Store.
Use open-source Maggy to write and debug PyTorch code on your local machine and run the code at scale without changing a single line in your program.
Recently, one of Sweden’s largest banks trained generative adversarial neural networks (GANs) using NVIDIA GPUs as part of its fraud and money-laundering prevention strategy.
Seeing how Redis is a popular open-source feature store with features significantly similar to RonDB, we compared the innards of RonDB’s multithreading architecture to the commercial Redis products.
Learn how to design and ingest features, browse existing features, create training datasets as DataFrames or as files on Azure Blob storage.
Connect the Hopsworks Feature Store to Amazon Redshift to transform your data into features to train models and make predictions.
Many developers believe S3 is the "end of file system history". It is impossible to build a file/object storage system on AWS that can compete with S3 on cost. But what if you could build on top of S3
Use JOINs for feature reuse to save on infrastructure and the number of feature pipelines needed to maintain models in production.
The feature store is a data warehouse of features for machine learning (ML). Architecturally, it differs from the traditional data warehouse in that it is a dual-database.
Hopsworks supports machine learning experiments to track and distribute ML for free and with a built-in TensorBoard.
Integrate with third-party security standards and take advantage from our project-based multi-tenancy model to host data in one single shared cluster.
This blog introduces the feature store as a new element in automotive machine learning (ML) systems and as a new data science tool and process for building and deploying better Machine learning models
Try out Maggy for hyperparameter optimization or ablation studies now on Hopsworks.ai to access a new way of writing machine learning applications.
Learn how to integrate Kubeflow with Hopsworks and take advantage of its Feature Store and scale-out deep learning capabilities.
We have many conversations with companies and organizations who are deciding between building their own feature store and buying one. We thought we would share our experience of building one.
Integrate AWS SageMaker with Hopsworks to manage, discover and use features for creating training datasets and for serving features to operational models.
This blog introduces the Hopsworks Feature Store for Databricks, and how it can accelerate and govern your model development and operations on Databricks.
Read how ExtremeEarth brings Large-scale AI to the Earth Observation Community with Hopsworks, the Data-intensive AI Platform.
Introducing the feature store which is a new data science tool for building and deploying better AI models in the gambling and casino business.
.svg)
.png)
.png)
.png)
.png)
