In our previous article, we discussed how predictive analytics can help Customer Success teams and why a good Customer Success Management (CSM) solution should have both predictive analytics and rule-based alerting capabilities.
In today's post, we lay out the steps involved in implementing machine learning models as well as what to consider when evaluating CSM solutions with predictive analytics.
We’ve described how machine learning leverages the data being collected by SaaS vendors in order to predict which customers need the attention of the Customer Success team.
It does this by analyzing historical data and learning how customers behaved in the past in order to predict how others might act in the future.
Let’s take a look at the steps involved in implementing machine learning models as background for what to expect, as well as what to consider when evaluating CSM solutions with predictive analytics.
Those steps include:
- Define the business goals
- Identify the data sources
- Feature extraction
- Model training and validation
- Model maintenance and refinement
Define the Business Goals
The first step when getting started with machine learning is to define your business goals. For most SaaS Customer Success teams, the goals include predicting one or more of the following:
- Which customers are at risk of churning?
- Which customers are likely to expand (up/cross-sell)?
- Which are likely to convert from a free product or trial to a paying account?
Make sure the CSM solution addresses all of your goals, including what you need now and in the future. Don't settle for a system that only predicts churn. You may not have an upsell product now, but you could in the future.
Identify the Data Sources
The next step is to identify the data sources to be used in the machine learning models. This shouldn’t be limited to “behavioral” data like product usage or support tickets. Different segments of customers may behave differently, so it’s important to include as much information as you have about your accounts (e.g. industry, revenue, # of employees).
At a minimum, if you classify your customers into tiers, this data should be included so the models can evaluate each tier for its own best signals.
In addition to the standard data feeds to a CSM system (e.g. product usage, CRM, billing, support ticketing), you should also consider any customer data you have that is specific to your business.
For example, if your business is a job search site, you may have data on the number of candidates interviewed for a specific job or the number of resumes a candidate sent out. In many cases, this type of custom account data can be very informative to a predictive model.
To make predictions about what customers will do in the future, you need to have good data on what they did in the past. If you have historical data, your CSM vendor can shorten the time to develop working models. Otherwise, you’ll have to start with rule-based alerts while you’re collecting enough data.
How long do you need to collect data before the models become useful?
It’s not how much time that matters, but whether there are enough examples of the outcomes you’re looking for (e.g. churned accounts, accounts who upgraded).
It may take weeks until the models have enough data to be useful, depending on what’s happening with your business.
CSM Solution Considerations
- CSM solutions should allow easy integration with any data, even if it's not stored in a standard 3rd party system such as a CRM.
- Make sure your CSM vendor is including as much data as possible in their models, and not limiting it to product usage.
This step involves preparing the data for modeling.
It starts with the initial set of raw data and creates derived values (called features) that can better inform the model. For example, you might have a user who consistently uses a capability in your product 100 times per week and one week she uses it 120 times. In absolute terms, this is a jump of 20 uses. Another user may have only been using it 3 times a week, and then one week they use it 6 times. While the absolute change of 3 uses is much smaller for the second user, the rate of change is significantly higher (100% increase vs. 20% increase).
In some situations, rate of change may be more useful as a predictor than the absolute change.
Data scientists creating models will look at many derived values from the raw data to see if they can achieve better accuracy from the model. By analyzing things like rate of change, ratios or momentum, they can extract more meaningful signals from the data. A common use involves normalizing data across accounts that may have significantly different numbers of users.
CSM Solution Considerations
- Your CSM vendor should talk with you to better understand your business and help them extract the most informative features. Be wary if the vendor claims they can build the model on their own or they’ll have it ready in a couple of days. This is data science, not dart throwing.
- CSM vendors who are exclusively analyzing “Customer Success” outcomes, like churn or up-sell, across lots of companies will have a better understanding of how to optimize a model as compared to general machine learning vendors who employ the technology for a wide variety of uses.
Model Training and Validation
Once the feature set has been created, the next step is to train and validate the model. This requires “labeled” data, which simply means that for each account in your data set, the outcome you’re trying to predict is known. If you are trying to predict churn, for example, you’ll need to identify which of the customers in the data set actually churned and which didn’t.
The CSM vendor will usually divide the data into two sets.
One will be used to train the model using a technique called supervised training. The rest of the data is used to test the model. This allows the vendor to estimate the accuracy of the model, and make adjustments as needed.
CSM Solution Considerations
- There are a variety of machine learning algorithms and techniques that can be applied to the feature set. The CSM vendor should try several to see which work best for your business. The vendor should provide you with a report on the accuracy of the different models.
- If you are looking to predict multiple outcomes (e.g. churn and upsell), the CSM vendor should set up separate models for each prediction. CSM solutions that offer a single “predictive health score” that predicts both churn and expansion are simplistic and ineffective.
- Make sure the vendor is creating models that are unique to your business – not just using a basic model that they use for all of their SaaS accounts.
Model Maintenance and Refinement
Ongoing model maintenance and refinement should be part of the CSM vendor’s standard offering.
Think of it as “data science-as-a-service.”
The vendor should have full responsibility for creating and maintaining the predictive models. There should be no requirement for you to have data scientists on staff.
Don’t be confused by CSM vendors who deliver predictive analytics as a separate product or service. Some vendors may offer to sell you professional services where they’ll analyze your data, and suggest better rules to be used for alerts. You’re paying extra for that service, and if you only use it on a quarterly or semi-annual basis your alerts won’t be as accurate as they should be.
Machine learning models should be updated on a regular basis with data being collected each day to generate new daily predictions.
CSM Solution Considerations
- Make sure the vendor’s predictive analysis is part of their standard CSM solution. You shouldn’t have to pay extra for this capability.
- The CSM vendor should meet with you from time to time to discuss any changes to your business that could impact the models, as well as provide regular updates on model accuracy.
Predictive analytics is a powerful technology whose use is growing rapidly. Customer Success software that incorporates machine learning allows you operationalize the technology by building it into your business processes.
By enabling the Customer Success team to proactively focus on accounts that are likely to churn, expand or convert, CSM solutions with predictive analytics can significantly improve the effectiveness and productivity of the CSM team. Customer Success Managers can prioritize their efforts on the accounts that most need their attention, and in so doing they are equipped to maximize customer retention, engagement and lifetime revenue.
Want to learn more? Grab our eBook: Predictive Analytics for Customer Success.