You are here

How to fix the technical interview (and maybe the developer shortage)

public://pictures/Mitch-Pronschinske-TechBeacon.jpg
Mitch Pronschinske, Evangelist, HashiCorp

The plague of academic and impractical, algorithm-centric technical interviews in the software development industry is a known problem, and one TechBeacon covered in the post "Developer shortage, or time to rethink the technical interview?" This post continues that conversation with examples from companies that have come up with better ways to interview developers, complete with do's and don'ts for conducting developer interviews. (I'm also interested in what you have to say about how to run developer interviews, so leave your comments below.)

Let's pick this conversation back up with a comment from an interviewer (allegedly from Google) firing back at a developer who was criticizing the algorithm-centric interview:

"I don’t have time to read your github projects, or your blog, or your existing code, or your meticulous attention to detail in your commit messages. I get a resume in, I look at it for 10 minutes, if it’s sane I set up a phone screen. ONLY to prove that you can pass the most basic coding test... Once we meet in person, you need to show me that you can code and articulate a design to a problem. The format sucks, sure. But you need to be prepared. I’m not going to spend 2 hours reading your blog for every candidate that I interview. I don’t have time for that."

Even this interviewer who defended algorithm-centric interviews admitted that the format sucks. Can this vibrant industry really not come up with a more innovative way to conduct the hiring and interview process?

Startups and small companies tend to be more innovative when it comes to developer interviews.

Buffer is one small company that is very innovative when it comes to managing and paying employees. Its hiring practices are also innovative when compared with the bureaucratic, Computer Science(CS)-ruled interviews of many development firms.

Buffer focuses mainly on culture fit with three culture interviews, and then it does contract periods for new hires. Rather than taking the candidate's claims at face value in an overly complex interview, it just confirms the culture fit and looks at the candidate's previous development work when judging technical aptitude. The real interview is a four-to-six-week contract period for candidates who make it through the initial stages.  

This is right in line with the research Laszlo Bock of Google cited when looking into the best indicators of future candidate success: honest-to-goodness units of real work from the company. Another advantage of contract periods is how quickly they help you discover if a candidate works well with your team. That's something you can't discover with certainty in an interview. 

I'll let the CTO explain why he gave up on the algorithm-centric interview:

"I remember asking [algorithm-centric] questions like these and slowly realizing they never really told me much about how good of a fit a candidate would be on our team. If anything, I’d weed out a candidate who hadn’t taken a computer science course (or paid attention in class), even though I knew they had the capability of learning those concepts after seeing what they’ve accomplished in side projects."—Sunil Sadasivan, CTO at Buffer

I didn't even think about that! You're putting candidates who didn't get a CS degree at a disadvantage even though they may have had to work harder (or work on more real-world projects) to get where they are. 

Here's another quote from the Buffer CTO:

"Asking technical questions doesn’t allow me to determine their ability to solve problems inline with how we approach things. We’ve been trying to build a team that follows Lean principles, that doesn’t try to re-invent the wheel, or spend time optimizing for perfect code. Getting rid of technical questions in interviews has been one of the best things I’ve done as it allowed me to focus on what really matters for us in an interview."Sunil Sadasivan, CTO at Buffer

If more companies agreed with Sunil and changed their interview tactics, would there still be an uproar about a skilled developer shortage?

While I'm sure there are plenty of shortages in positions that require a heavy amount of expertise in niche fields, the developer shortage in general web development and software engineering fields may be self-inflicted by the software industry's hiring practices and misconceptions.

Right in my own backyard of Durham, N.C., there's a Django and JavaScript front-end shop called Caktus that also eschews algorithm quizzes, coding tests, and puzzlers in its interviews. I listened to Caktus' technical director, Mark Lavin, speak at a local conference last year about its main goals in hiring, which are:

  1. Finding a candidate who can add to the team's overall knowledge profile (not someone who knows all the same things and thinks the same way as everyone else)
  2. Picking someone who is not a rockstar, but someone who can complement the team well and make it stronger.

Dev celebrity Scott Hanselman agrees: "The rockstar developer is a myth, the rockstar team is reality." You can hire a team of "average" people (most people are average, mathematically speaking) who—when they're working well together—can outperform teams with above-average people.

But hold on a second. Buffer and Caktus are smaller companies. They don't have to hire as many developers. Is interview innovation a luxury that bigger companies can't afford?

I know both of these companies are smaller and more nimble and don't require people who can work on anything, but it's surprising that the supposedly innovative tech giants can't come up with better ways to do technical interviews that still scale to meet their needs.

Off the top of my head, I can think of a few things recruiters could do at scale that would be much more illustrative of a developer's abilities than algorithm questions: 

  • Use tools to analyze the quality of code samples
  • Find out how widely a candidate’s code was included as a dependency in a package ecosystem such as npm (that’s how an academic researcher’s work would be judged)
  • Give candidates a take-home, realistic (or real) work sample project

While startups snatch up talented developers with lots of potential and others move into freelance, how can these companies afford not to figure this thing out?

Standardized processes create standardized developers, and right now that standard is white and Asian males.

Oh, so now you're bringing up diversity?

Diversity is a huge problem in the software industry. I could write a whole other article on diversity in tech.

Caktus' style of hiring, which has identified great developers without algorithm tests, has also given it a level of employee diversity that most tech companies only dream of. 

So the theme seems to be that big companies are missing out on potentially great, more diverse developers?

Yes. It all comes back to that.

Some will argue that algorithm questions and brainteasers are an observation study. Essentially they're admitting that it's a ruse that isn't meant to test your true knowledge, but rather to see how you perform under pressure. Can you talk your way through a problem? Do you ask questions? That kind of information is actually good to know, but it needs to be done in the right way. Going overboard on the pressure can have negative consequences and filter out potentially good candidates.

Like, maybe, introverts?

Yes, and possibly other minorities in software development that might already feel as if the odds are stacked against them in this space.

Diversity is important, and not just in race and gender, because you also want diverse personality types. You mentioned introverts, and you're correct that they are traditionally at a disadvantage in these interviews. Companies should think about ways to keep the playing field even, because filtering out all the introverts, for example, leaves a powerful personality type out of your team.

One thing Caktus did to make people feel more comfortable applying for their positions was make a simple change to how they wrote its job postings. It was a simple rethink: Post job ads that actually described what new hires would do and what performance milestones they would attempt to meet in their first 30, 60, and 90 days. No more exorbitant experience required or keyword soup postings.

It sounds so easy! Why haven't we come up with better ideas like this for every single step of the hiring process?

I think résumés/CVs and cover letters are two other ancient rituals that need a revolution throughout the software industry and beyond. Right now, they're just used by HR system filters that throw out any résumés that don't have the correct combination of keywords.

Each new generation of developers is getting increasingly fed up with stale hiring practices, as evidenced by a recent survey by Devpost, which shows that student developers hate cover letters, prefer take-home coding projects, and find today's interviews to be "stressful and unproductive." Most students would rather talk about their projects and experience in interviews instead of answering computer science questions. There's agreement with this viewpoint from the interviewer side too, as we can clearly see from the practices of Buffer and Caktus.

I'll close with some resources from a third engineering firm that has recently innovated around the developer interview because it also thinks the state of technical interviews is broken—the blogging network Medium.

Just a few weeks ago, Medium released detailed documentation on its hiring practices overhaul. Its new style of engineering interviews is available on three web pages under the Creative Commons Attribution Share-Alike license. The sections include:

If you are looking for a detailed instruction manual on how to build a better technical interview rubric, these documents from Medium are a great framework to start with. But before you read them, educate yourself on the fundamentals of good developer interviews with Stack Overflow's guide, "How to Interview Developers."

Looks like some companies, even many of the ones we consider leaders in our industry, have a lot of catching up to do. What should companies take away from this conversation?

I'll boil it down to a list of dos and don'ts

  • Don't ask brainteasers, puzzlers, or riddles.
     
  • Don't make the interview algorithm-centric. Don't bring up any college algorithm- or CS theory-style questions if they're not completely relevant to the candidate's day-to-day work.
     
  • Don't ask question after question after question. Have a two-way conversation.
     
  • Don't have a whole interview focused on one coding exercise or question. You can only demonstrate so much knowledge with one exercise.
     
  • Don't use HR for the in-person interviews. Have the developers and leaders on your team (preferably the best ones) interview the candidate for the crucial stages.
     
  • Do your homework on the candidates. If you don't have time, narrow down the candidates until the number is manageable. Take just 15 to 30 minutes to look at a candidate's code.
     
  • Do ask for a code sample (just as you would for a writer) or have candidates complete a simple online programming test. This is just a sanity check to see if they can do basic programming.
     
  • Do research on the four different types of interviewer bias and try to eliminate them.
     
  • Do give candidates a piece of actual, honest-to-goodness work for your team. Do pay them for it.
     
  • Do have them discuss or give a presentation (with a whiteboard is fine!) on a coding project that they have worked on and they're passionate about. Find out why they design things the way they do.
     
  • Do a four-to-six-week contract period for new hires.
     
  • Watch this video.

And what can developers do to avoid algorithm-centric interviews or get the software industry at large to stop using them? 

If you're not desperate for a job, start by telling recruiters up front that you don’t do algorithm trivia interviews. Maybe with enough pressure from the developer workforce, employers will start to notice that their interviews aren't getting them the best candidates or the best teams. But it's hard to change old habits, especially when they're the habits of a large percentage of the industry. Some executives will continue to blame it on The Developer Shortage™.

There's hope, though. For a while, brainteasers were the norm at many big companies that wanted to be like Google. Those have since been widely denounced (mainly by Google itself) and died out throughout much of the industry. With enough pressure and attention, we can kill algorithm-centric interviews, too.