AI projects face new challenges, need new roles, and engage in new activities as they move into mass commercialization

I wrote previously about the emergence of applied, agile, assured AI. This post is a deeper dive, explaining the diagram, below, in more detail:

Over the years, computer science has evolved through four eras, which are playing out as well, with some delay, in data science (my emphasis here is the AI/ML/DI side of data science, because the pure data side is shared with CS):

1. R&D: Here, the goal was to create CS to start with and, ultimately, to prove that the technology was workable. Most of the people who did this were computer scientists, working in academic institutions.
2. Core algorithms: When I was a CS professor in the 90s, we taught how to write core algorithms: search, sort, data structures basics, and the like. The challenge we faced was to create the core building blocks of the discipline, to enable the next phase of development.
3. Commercialization: During this phase, the emphasis shifted towards mass adoption. The challenges were new as well: the emphasis was less on creating the fastest, smallest, most elegant code, and more on managing end-to-end projects such that they met business needs, were delivered on time, were user friendly, and—let’s face it—didn’t fail. AI/ML/DI are on the verge of this phase today.
4. Agility / ubiquity: The advent of the PC marked the start of an age of computer ubiquity. I remember when I was first coding in the 1970s I had to explain the difference between software and hardware to my relatives and friends. No more: we all carry massive compute capabilities in our pockets, and this technology is part of our everyday lives. For technologists, the task is less about creating core methods than assembling existing libraries in novel ways to solve problems. This is the near future of AI/ML/DI, for better or worse.

As data science is crossing into commercialization, we’re seeing new activities, roles, and challenges emerge. As with CS, activities receiving new emphasis include matching technology to business needs, and including engineering disciplines like quality assurance. Additional activities are also emerging, including data preparation for AI in such a way that it avoids bias and unintended consequences, and activities like those we outline in our Agile AI methodology.

If you’re looking to get ahead of the curve in your career in data science, you might look to the roles that emerged during the commercialization of CS, but with an AI twist: consider AI-specific QA, devops, project management, decision analysis, and more.

And, probably most fundamentally, our primary challenge is no longer to advance the state-of-the-art. It is, rather, the often much harder problem of ensuring return on investment with on-time, on-budget, delivery of projects that succeed.

We’re wiser than we were, though, and so we are defining “success” more broadly. We know today that technology can have unintended consequences. This is especially true for AI, which is often deployed as a “black box”, which humans employ as a deniability smoke screen (“I don’t know why, the algorithm made me do it”). So we must take responsibility for success at a broader level now, maximizing the value while minimizing the risks as we move into ubiquity.

The bottom line: data science (/AI/ML/DI) is becoming an engineering discipline, just as CS did as software engineering emerged in response to the need for reliable, trustworthy systems. Looking back to other engineering disciplines gives us important clues as to how to maximize data science success.

Image by mohamed Hassan from Pixabay