Reliable Machine Learning: Applying SRE Principles to ML in Production

This means that we talk about the messy, complicated, and occasionally frustrating work involved in shepherding data correctly and responsibly; reliable model building; ensuring a smooth (and reversible) path to production; safety in updating; and concerns about cost, performance, business goals, and organizational structure. We attempt to cover everything involved in having ML happen reliably in your organization.

Why We Wrote This Book

We firmly believe at least some of the hype: ML and AI techniques are currently reshaping computing and society at an accelerating rate. To that extent, the public hype has not caught up with the private reality in some respects.1 But we are also grounded and experienced enough to understand just how laughably unreliable and problematic many real-world ML systems actually are. The technology press writes about space flight, while most organizations still have trouble staying upright on their bicycles; these are the early days still. Now is the perfect time to actively pay attention to what ML can do and how your organization might benefit from it.

Having said this, though, we recognize that many organizations are worried about “missing out” on ML, and everything it could do for (and to) their organization. The good news is, there’s no need to panic—it is possible to get started now and to be sensible and disciplined about how you work with ML, in a way that successfully balances both obligation and reward. The bad news, and the reason many organizations are worried, is that the curve of complexity is quite steep. Once you get past the simpler aspects, many of the techniques and technologies are just being invented, and it’s hard to find a solid, paved path.

This book should help you navigate that complexity. We believe that, despite the immaturity of the industry, there is much to be gained by focusing on simplicity and standardization, an approach that has the beneficial side effect of making it easier to get started. Ultimately, organizations that deeply integrate ML into their business will benefit—some substantially2—but they will, of course, need a degree of sophistication about how that is done. A simpler, standardized foundation will facilitate developing that capability better than ad hoc experiments, or even worse, a system that works but no one knows how or why.

Intended Audience

We are writing for anyone who wants to take ML into the real world and make a difference in their organization. Accordingly, this book is for data scientists and ML engineers, for software engineers and site reliability engineers, and for organizational decision makers—even nontechnical ones, although parts of the book are quite technical:

Data scientists and ML engineers: We’ll explore how the data, features, and model architecture you use change the way your model works, and how manageable it is in the long run, all with an eye to model velocity.

Software engineering building ML infrastructure or integrating ML into existing products: We address both how to integrate ML into systems and how to write ML infrastructure. An improved understanding of how the ML lifecycle works helps with developing functionality, designing application programming interfaces (APIs), and supporting customers.

Site reliability engineers: We’ll show how ML systems typically break and how best to build (and manage) them to avoid those failure modes. We’ll also explore the implications of ML model quality not being something a reliability engineer can entirely ignore.

Organizational leaders who want to add ML to their existing products or services: We will help you understand how best to integrate ML into your existing products and services, and the structures and organizational patterns required. Having a sensible way of assessing risks and advantages when making ML-related decisions is important.

Everyone who is rightfully concerned about the ethical, legal, and privacy implications of developing and deploying ML: We will lay out the issues clearly and point to practical steps you can take to address these concerns before they cause damage to your users or your organization.