From BI to AI: A Data-Driven Journey

We have all heard the clichés about today’s data industry. Data is the new oil. The new gold even. Enterprises large and small understand that they in fact live on data. Data drives processes every day - and as the digitalisation of our economy progresses, data literally becomes the true lifeblood. 

Having worked in the data industry for the past few decades, we feel like we have had a front row seat in this evolution, and strongly feel that there is no end to the range of new possibilities in the world of data. New data platforms have emerged, and new applications have been enabled by these platforms. 

Some of these newer applications are based on new ways of dealing with massive amounts of structured, semi-structured and unstructured data. This is new: traditional technologies were always engineered to provide “Business Intelligence” (BI) - specifically helping business leaders to better understand what had been and was currently going on in their organizations. Now “Artificial Intelligence” (AI), more specifically Machine Learning (ML), is tasked with a different assignment: to predict the future.

This is a different task from before: 

• Data is everywhere - in all kinds of shapes and sizes. We need to ask ourselves the question of how to get a holistic understanding of our organization by leveraging all data.
• Machine learning systems promise to bring data-based decisions to every part of every enterprise. No need to rely on hunches anymore, as we now have data, and we can process it (semi-)automatically. Machine Learning systems allow us to build better models of our data as we train them more with bigger and better data. 
• As these models get more refined, businesses will want to use these techniques to more reliably move from the analysis of historical post-factum understanding, to forward-looking, predictive analytics: Machine Learning and Artificial Intelligence are making a real difference and adding tremendous value to corporate data infrastructures, as the automated nature of the model creation and improvement allows us to make higher quality, learned projections/predictions about the potential outcomes of our operations based on historical (training) data. In short: predictive analytics will get a massive boost because of Machine Learning.
• The next step is now for those systems to give us the means to embed the decisions and predictions: allowing the products and services themselves to be more adapted, personalized, and responsive to the users and their demand. 

It may be useful to contrast this approach with previous analytical approaches, like for example the one taken by traditional Business Intelligence tool sets:

As you can see Machine Learning requires different types of tooling that is very complementary to traditional analytical toolsets. “Business Intelligence” tools have been characterized as using traditional algorithms and statistics that were mostly used on smaller historical, and often well-structured datasets. This provided a less refined model of that data - and therefore less accurate predictions - except in simple cases. 

Machine learning tools are different:

• They are built with Model-centric automatability in mind, and can be used to build systems quite easily. We like to think of “FTI Pipelines”, and think they are key. They cover the following steps in Machine Learning systems:‍
  
  - Feature engineering, which will first power the:
  - Training of our Machine Learning model, and then also be used for:
  - Inferencing or predicting the outcomes of future states. 

• Many of these tools are available as separate components, in one way or the other. We find that Open Source Software is often leading the way, even though some proprietary tools of course exist.
• Most of these tools are not easy to use - they basically require you to be a programmer of some sorts.
• Many of these tools take significant effort to integrate and connect into a coherent architecture.

At Hopsworks, our conclusion has been that this may be ok for building AI/ML prototypes and test beds, but if we want to make production quality AI/ML systems, then this becomes a real challenge that just slows down adoption and implementation of the technology.

Therefore, Hopsworks has made it its mission to offer products and services that will help with operationalizing AI and ML. This will require a different approach and different platforms, like for example a feature store. Some people have called feature stores the “data warehouse” for your Machine Learning data. It will allow for all your Machine Learning operational processes to be built around it, and allow you to get to market more quickly. This is the vision and architecture of Hopsworks in a nutshell - and we look forward to making a difference in this exciting space!