How to hire a machine learning consultant
More and more organizations are realizing the tremendous benefit of machine learning to their bottom line, yet many are not ready to hire a full-time machine learning expert. So a machine learning contractor/consultant/freelancer makes sense.
For instance, your business might depend upon “just in time” delivery of some goods, and you need to predict when customers will run out. Or your municipality might be looking to understand what decisions will lead to your sustainability goals. Or you want to learn the signals that customers send when you’re about to lose them. Or build a visual recognition system to detect faulty parts on your assembly line.
For the first time in the last year or so, the decreasing price of computing resources, coupled with an explosion in machine learning performance, means that there’s a great potential for business outcomes that comes from applying machine learning to your data.
If you’re considering this route, there are a few important principles to keep in mind.
Before the project
Most of my clients start out with a pretty clear idea of what the machine learning project should produce. I’ve found, however, that a few important questions often go unanswered. Be sure to figure these out ahead of time.
- Are we looking for a person to develop an entirely new machine learning algorithm, or to apply an existing algorithm to our problem? This is like asking “Do you need someone to design a new car, or to drive an existing one?” Since many machine learning experts come from academic settings where the focus for many years has been on designing new machine learning algorithms in the first place, the skill of “driving” has been a bit overlooked. So be sure to look for a consultant who knows what it means to match the right algorithm to your problem, and then to do the data management, parameter tuning, and training work to get it built.
- How does this machine learning algorithm fit into our business as a whole? A clear understanding of your business context—often expressed as a process or data flow—helps your consultant to know whether to tune for true positives or true negatives, how often new data is available, what performance level is “good enough”, and more.
- Is this a fully automated or human-in-the-loop situation? If your system will be processing thousands of data items every second, without human intervention, then you’re going to build a very different system than if a key requirement is that humans can understand (and so override or supplement) its logic. If the latter, then that’s what decision intelligence is all about.
- How far towards deployment do we want our contractor to go? They can just build the core algorithm, and deliver it in R (or SAS or SPSS), which might then need to be converted to a deployment language by your developers, or they can do that conversion, embed the algorithm in an application, and even go as far as to develop and deploy the application in a big data environment. There are different areas of tool expertise required for each step of this process, and it’s rare to find someone expert in all of them.
- Is our data ready? Is your consultant responsible for cleansing, validating, etc.? Some consultants, especially those who come from a data science or database background, are quite good at this. Some not so much. Most machine learning systems use quite simple data formats, such as a single .csv file, so you’ll need to get it ready for them. Usually, I’ve found that data preparation is more cost-effective to do with data management expert resources than a machine learning person.
Typical project steps
A typical machine learning project follows some subset of the following steps:
- Agree to how success will be measured. The simplest here is a simple holdout test set. Lately, though, I’ve had a few clients who say that success also depends on how understandable the system behavior is (e.g. recognizable visual features or a decision tree that makes sense and can be trusted). Since most good machine learning algorithms are “black box”, it’s good to ask.
- Select the machine learning algorithm to use, based on the problem. The most common algorithm categories are classification, regression, and recommendation. If needed, develop a new algorithm. But keep in mind, this can be expensive!
- Agree to how the algorithm will be tuned. For true-positives or true-negatives? Top-10 better than baseline top-10? Straight sum of squared errors?
- Agree to any feature engineering. This step is often overlooked, yet good feature engineering can make a big difference to performance. Though there have certainly been successful projects using “raw” data, your subject-matter expertise in how the data might be modified to best find patterns can really help. Your machine learning expert should be able to collaborate with you on this step, helping to elicit your expertise in this regard.
- Prepare the data. As above, often this means converting multiple tables to just a single one. The “trap for young players” here is that data preparation time is often much larger than originally expected. I recommend you make a careful estimate, then double it.
- Choose the machine learning platform tool.
- Run a baseline. This is an important step that’s often overlooked. I’ve often found that, especially with good feature engineering, a simple classifier can produce great performance. This is also a good way to test your success measurement method.
- Do the learning. This is the core of the project. Pivot and grid search as needed on parameters, classifier size, and even algorithm, as needed.
- When done, report results. If possible, produce a fabulous visualization of the results. If you’re ambitious, visualize learning in process and/or the inside representations.
- Integrate the algorithm into a production environment. In many circumstances, this involves re-implementing it in code (e.g. if it was originally developed in R or SAS).
- Integrate the algorithm into a “big data” production environment, using cluster, GPU, or other high-performance compute and data storage / management resources.
Now, I’ve never had a project that required all of these steps. A typical contract is around three to five of them. So treat the list as a menu of options as you decide the scope of your project.
Update: check out this MCubed Keynote with the 10 biggest lessons learned for success in an applied machine learning project.
(PS drop me a line below if you’d like to talk about how I can help you, too.)
