The pipeline was the product

Before the COVID-19 pandemic, I started teaching myself Python by building a lottery predictor. What began as a scraper and a local training process grew, over several iterations, into a tested event-driven MLOps system.

I learned early that the predictor would never be a viable product. Lottery draws do not contain a signal that a better model can uncover. I kept working on it because that failure exposed a more interesting problem: what remains valuable when the original goal fails but the engineering around it becomes reusable?

The premise failed before the system did

The first challenge was obtaining data I could trust. Published results were inconsistent enough that collection could not be a one-off script. The scraper had to preserve progress, tolerate missing draw identifiers, normalize values, and fail visibly when the source changed.

The resulting dataset contained roughly 2,400 historical draws. It was small and multivariate. There was nowhere for a sophisticated model to hide weak evidence.

I worked through recurrent and convolutional approaches before moving to a Transformer-based design. The models changed; the underlying constraint did not. A loss curve could improve while the evaluation remained invalid, and a plausible prediction could still be indistinguishable from chance.

That made evaluation more important than novelty. The pipeline used chronological partitions with a gap between them, and preprocessing was fitted on training data only. Mean and last-value baselines sat beside the model’s MAE and RMSE. A “best candidate” score remained diagnostic rather than becoming the headline result.

One iteration produced several candidates through independent prediction heads. When they collapsed toward the same output, I tested Monte Carlo dropout, different dropout rates, Gaussian noise, and a diversity objective. The outputs became different without becoming meaningfully dependent on the input.

That distinction ended the argument. Variation was not information. Under squared error, every head had a reason to converge on the same conditional mean. Quantile regression described the uncertainty more coherently, but it did not make the lottery more predictable.

Uncertainty estimates made the model’s limits clearer. They did not change what the data could support.

The discipline was not in finding a flattering metric. It was in preserving a result that could disprove the premise.

Change the objective when the evidence changes

The first version ran on hardware I managed myself. That was useful while I was learning the mechanics, but eventually the environment became a project of its own. Keeping drivers, libraries, and training dependencies compatible consumed attention without improving the experiment.

Moving the next iteration to Paperspace was a small but important decision: identify the constraint consuming attention and move it behind a better boundary. Managed compute removed one distraction and exposed the larger gaps around training.

By then, the model question had an answer. Continuing to optimize it would not change the product outcome. I changed the objective instead. The project would be evaluated as a machine-learning system: could it make a run reproducible, automate its lifecycle, control cost, expose failure, and retain enough evidence to explain what happened?

This was not a way to declare the original idea successful. It was a decision to stop spending effort on a closed question and use the remaining work to develop a reusable capability.

Make the system produce evidence

Training became a declared contract: validate configuration and data, create leakage-aware partitions, fit preprocessing, train, evaluate against held-out data, and write the checkpoint, scaler, metrics, plots, and completion event that described the run.

The AWS design placed that contract inside one inspectable lifecycle. New data in S3 initiated orchestration through EventBridge and Step Functions. Temporary GPU-backed Spot capacity performed training. A validation Lambda had to load the resulting checkpoint before inference could use it. Artifacts returned to durable storage; the worker terminated.

The automation did not improve the model. It made incomplete training visible, prevented inference from accepting a broken artifact, and bounded the cost of occasional GPU work. The worker could disappear; its evidence could not.

That boundary connected several lessons that had previously seemed separate. A checkpoint without its fitted scaler is not deployable. A metric without its configuration and data lineage is not an audit trail. A prediction that is never compared with the outcome invites exaggeration.

IaC made the infrastructure and IAM boundaries reviewable. Structured events and CloudWatch signals made failure observable. A small FastAPI dashboard kept predictions beside the results that followed. Property-based tests pushed the important invariants beyond their happy paths: chronological splits, training-only preprocessing, stable tensor shapes, consistent metrics, and rejection of malformed input.

Each addition closed a way the system could otherwise misrepresent what it saw, which artifact it used, whether a run completed, what it cost, or how well it performed. The training path, cloud handlers, infrastructure definitions, dashboard, and failure conditions were tried and tested against the same lifecycle.

Know when the answer is enough

The project produced two independent conclusions. The model had no viable forecasting signal. The system around it had become a credible account of how I would operate a machine-learning workload.

Continuing to tune the model would have polished a demonstration without changing either conclusion. Deprecating it was therefore not an admission that the work had no value. It was the final application of the discipline the project had taken years to develop.

The value of an experiment is not limited to whether it succeeds. It should be able to challenge its premise, clarify the next decision, preserve what remains useful, and show when further work will not change the conclusion.