Reference calls are a valuable but tricky hiring tool
Reference calls are a common part of any team’s hiring toolkit. After a positive interview round with a candidate, it’s common to consult with (hopefully) impartial third parties to verify strengths, clarify weaknesses, or simply provide an additional perspective.
These conversations provide a dimension that you can’t get from most interview processes. You get 3-6 hours with most candidates during a typical set of interviews, and these interviews will differ meaningfully from a normal workday. A candidate’s references have years of real-world experience that they can speak to and are invaluable for roles such as product management, marketing, or executive leadership where it’s harder to quantify hard skills or judge work samples.
References come in two main flavors:
- Backchannel: for example, when someone on your team or in your network worked with a candidate in the past.
- Candidate supplied: references that a candidate provides, who can speak to their specific skills, character, and experience. Often managers or senior peers.
Backchannel References
Backchannel references can be really valuable. If someone is a big jerk, or really smart, or an excellent leader, it will likely come out. They’re also a good source of unbiased information, which is at a premium given that candidate-supplied references are almost always biased positively (more on that below).
But backchannel references are inherently random and unreliable. Some candidates will naturally be close to your network and have significant potential for backchannel reference checking; some candidates won’t. Additionally, backchannel references that worked directly with a candidate are fairly rare – how much stock do you actually put in a salesperson’s opinion of a software engineer, and vice versa?
Quality backchannels are excellent, but only when relevant and available. Value them highly when they worked closely with a candidate.
Note that in many US jurisdictions you must get candidate consent before pursuing any backchannel references. Check the laws!
What’s Good Here?
References supplied by a candidate are much more common. They’re also more likely to be relevant as they typically worked closely with the candidate in question.
What’s good here?
But candidate-supplied references are kind of like asking “What’s good here” at a nice restaurant. You’re getting an opinion from someone with expertise that you lack, but they have absolutely no incentive to be anything but positive in their descriptions. Additionally, they don’t know you or what you need, at all.
The vast majority of candidate-supplied references are overwhelmingly positive – so much so that most contain very little signal. It takes an exceptionally foolish person to provide a reference who won’t describe them as a superhero. People also have little to gain and a lot to lose by speaking ill of past coworkers, so these calls are typically glowingly optimistic. Outright dishonesty is rare but positive spin is standard.
As a result, the trick to reference calls is figuring out how to divine true signal from a rose-tinted portrayal of the candidate.
Getting Signal From Positive References
Given this positivity bias, it’s important to ask the right types of questions. Here are a few ideas for a hypothetical candidate named Bob.
Aim for objective assessments that force references to be unambiguous. Binary questions are perfect here:
- Would you hire Bob again? (if yes, good)
- If the opportunity arose due to other circumstances, could you see yourself working for Bob in the future? (if yes, really great)
- Was Bob always respectful and professional to their teammates?
- Did Bob leave your team on good terms?
Another area where references can help with a hiring decision is by revealing negative space that you wouldn’t otherwise cover in an interview. Targeted but open-ended questions work best:
- If we hired Bob and you learned in 6 months that we’d parted ways, what do you think the most likely reason would be?
- What type of environment would Bob thrive in?
- What advantages, if any, does Bob bring to the table that would be hard for us to learn from the interview process?
- What else should we know about Bob?
(Many of these open-ended questions also don’t have obvious right or wrong answers, so it’s harder for references to game them to help out a friend)
And finally, if you want an assessment of a candidate’s overall aptitude, try questions that force references to rank candidates on a relative scale. Most candidate-supplied references are looking for ways to speak positively, and relative measurements allow them to say nice things while still giving you a general sense of their strengths. Examples include:
- What percentile rating would you place Bob at for their role? (In practice, no referral ranks anyone below around the 60th percentile, but this is fine – you can reset your scale to 60-100 and go forward from there)
- How did Bob rank compared to the other engineers/designers/salespeople/whatever on their team? Were they the best, the second best, the worst? (Most references will give you additional context for any rankings other than “best,” which is helpful for calibration)
Pay Attention to Negative References
If a candidate can’t supply any references from a particular company, that typically means that things didn’t go well. Most people are able to get at least one reference that they trust from a prior job. This isn’t a dealbreaker, but it should be taken into consideration as a potential red flag. Note also that candidates typically can’t get references from their current employer.
Additionally, if a reference provides an overall neutral or even negative review, you should take it very seriously. With the natural bias towards positivity, it’s rare and telling if a reference comes out negative.
Negative references are an imperfect but valuable filter. Not all toxic assholes will be revealed as such by testimonials, but if someone is reported to be a toxic asshole, in my experience it’s likely to be true.
Ultra-Positive References
Occasionally, a reference will absolutely rave about a candidate – “this is the best person I’ve ever worked with” etc. This is obviously a great positive signal, but you should take into account the reference’s context and standards.
For example, a reference who’s only worked at less successful or slower growth companies may have very different standards from someone who’s gone from Facebook => Uber => Stripe and has worked with some of the strongest people in the industry (although you never know). It’s important to get a sense for a reference’s own strength and perspective. When it comes to generalized measures of quality (“this person is a rockstar”), benchmark your assessments according to the speaker.
Takeaways
Getting good signal from references can be an arcane art due to the inherent positivity bias of candidate-supplied references, so try the following to make the most of these conversations:
- Consider backchannel references when viable.
- Remember that candidate-supplied references will be overwhelmingly positive and weigh them accordingly.
- Get more signal from references by asking binary questions, targeted open-ended questions, and questions that force the reference to rank candidates on a relative basis.
- Pay attention to negative or missing references.
- Gauge the background and experience of very strong references.