From Horse Racing calculation to Anti-Cheat Lookups

We have been working with Paddy Power, an Irish gambling company, to help them calculate odds for horse racing by using two different models: one for All Weather races and one for Flat races. Paddy Power also uses Hopsworks with batch predictions as an anti-cheating system making sure that no user “knows more than the betting company”. 

The Aim: 

• Data scientists were unable to easily discover and experiment with existing features and pipelines
• Sharing features across models was not possible
• Difficulties re-using features between the all weather model and the flat racing model
• Infrastructure depending heavily on a small and dedicated team to maintain it
• The data warehouse did not provide feature statistics or metadata, slowing down the process of feature engineering
• Python, the preferred programming language choice of most data scientists, is not supported in Redshift
• No centralized storage/sharing of features
• Maintenance issues
• Lacked the ability to collaborate

Why Hopsworks? 

• They integrated the Hopsworks Feature Store as a repository of features ready to be used for training models with the existing AWS SageMaker architecture. 
• Data scientists and analysts can now browse available features, inspect their metadata, investigate pre-computed statistics, and preview sample feature values. 
• Hopsworks also allows better centralization and accessibility of data as well as collaboration between teams.

Results: 

‍Improved Feature Quality

‍Improved models that generate more revenue

Faster Feature Engineering

‍Access to statistics and metadata, decreasing the time to generate training datasets

Exploratory Data Analysis

‍Discover pre-computed features, types of those features, descriptive statistics and the distribution of feature values

Feature Reusability

Previously engineered and quality-assured features become available to be reused - ready for training

Consolidated Feature Engineering Pipelines

Feature engineering code is not duplicated in applications, instead a single pipeline computes features for serving and training

Faster Models to Production

Data scientists can concentrate on improving models, and not on complex infrastructure for ensuring training and serving pipelines are kept in sync