Citizen
real-time public safety app
The training pipeline cost more than the models were worth
What I found
Every training run spun up a Dataproc cluster to prepare the data. One-hot encoding, feature assembly, repartitioning Parquet, all before a single model started learning. Feature logic lived in PySpark, disconnected from the code that served predictions to users. So the features a model trained on were never guaranteed to be the features it saw in production.
What I decided
Move the feature computation upstream into BigQuery as plain SQL, register those tables as Feature Groups in Vertex AI, and let point in time lookups replace the Spark transformations. Training and serving now read the same definitions. The Dataproc cluster was switched off entirely.
What it changed
Training became a BigQuery export. Feature drift between training and inference stopped being possible by design.
What runs today
Resultaat (aanvullen)