I don’t think anyone disputes that, it’s just that nobody has come up with anything better.
Take home exercises were a potentially better option (though they definitely have other big downsides) but they aren’t a sensible choice in the age of AI.
Just taking people’s word for it is clearly worse.
Asking to see people’s open source code is unfair to people who don’t have any.
The only other option I’ve heard - which I quite like the sound of but haven’t had a chance to try - is to get candidates to do “live debugging” on a real world bug. But I expect that would draw exactly the same criticisms as live coding interviews do.
I’m not sure that offline or alone coding tests are any better. A good coding interview should be about a lot more than just seeing if they produce well structured and optimal code. It’s about seeing what kinds of questions they’ll ask, what kind of alternatives and trade offs they’ll consider, probing some of the decisions they make. All the stuff that goes into being a good SWE, which you can demonstrate even if you’re having trouble coming up with the optimal solution to this particular problem.
… that’s why you do a follow up interview and review their code, and maybe leave some things a little ambiguous to see if they ask you questions (telling them it’s okay to email questions and mostly expected)
Why did you decide to do ABC this way? What do you think about having done it XYZ way instead?
I know you didn’t have time to write a full test suite, but what areas of what you wrote would be best to focus on tests and why?
You can ask them so many things about what they wrote.
That’s like… how it works in the real world. They ask questions to product as they come up, they get questioned on their work in code reviews
Unless you work somewhere where you pair code 100% of the time anyway…
If you just look at it as a pass or fail and are not doing a detailed review with them after, you’re doing it wrong.
Sure you can move some parts of the conversation to a review session, though I think the answers will be heavily influenced by hindsight at that point. For example, hearing about dead end paths they considered can be very informative in a way that I think candidates assume is negative. Nobody expects you to get it right the first time and telling the interviewer about your binary tree solution (that actually doesn’t work) can be a good thing.
But the biggest problem I think with not being in the room as an interviewer is that you lose the opportunity to hint and direct the candidate away from unproductive solutions or use of time. There are people who won’t ask questions about things that are ambiguous or they’ll misinterpret the program and that shouldn’t be a deal breaker.
Usually it only takes a very subtle nudge to get things back on track, otherwise you wind up getting a solution that’s not at all what you’re looking for (and more importantly, doesn’t demonstrate the knowledge you’re looking for). Or maybe you wind up with barely a solution because the candidate spent most of their time spinning their wheels. A good portion of the questions I ask during an interview serve this purpose of keeping the focus of the candidate on the right things.
hey, I love this idea, but we tried it and we kept getting candidates who managed to BS their way onsite and then waste our time ultimately.
it just didn’t work. I really want it to work because I hated live coding too but it just didn’t.
you can make live coding interviews that aren’t actually difficult questions and are more about showing that you can think and write the most basic of code. that’s what we do now.
The article isn’t saying don’t check, it’s saying that live coding interviews are a bad measure.
I don’t think anyone disputes that, it’s just that nobody has come up with anything better.
Take home exercises were a potentially better option (though they definitely have other big downsides) but they aren’t a sensible choice in the age of AI.
Just taking people’s word for it is clearly worse.
Asking to see people’s open source code is unfair to people who don’t have any.
The only other option I’ve heard - which I quite like the sound of but haven’t had a chance to try - is to get candidates to do “live debugging” on a real world bug. But I expect that would draw exactly the same criticisms as live coding interviews do.
What would you do?
I’m not sure that offline or alone coding tests are any better. A good coding interview should be about a lot more than just seeing if they produce well structured and optimal code. It’s about seeing what kinds of questions they’ll ask, what kind of alternatives and trade offs they’ll consider, probing some of the decisions they make. All the stuff that goes into being a good SWE, which you can demonstrate even if you’re having trouble coming up with the optimal solution to this particular problem.
… that’s why you do a follow up interview and review their code, and maybe leave some things a little ambiguous to see if they ask you questions (telling them it’s okay to email questions and mostly expected)
Why did you decide to do ABC this way? What do you think about having done it XYZ way instead?
I know you didn’t have time to write a full test suite, but what areas of what you wrote would be best to focus on tests and why?
You can ask them so many things about what they wrote.
That’s like… how it works in the real world. They ask questions to product as they come up, they get questioned on their work in code reviews
Unless you work somewhere where you pair code 100% of the time anyway…
If you just look at it as a pass or fail and are not doing a detailed review with them after, you’re doing it wrong.
Sure you can move some parts of the conversation to a review session, though I think the answers will be heavily influenced by hindsight at that point. For example, hearing about dead end paths they considered can be very informative in a way that I think candidates assume is negative. Nobody expects you to get it right the first time and telling the interviewer about your binary tree solution (that actually doesn’t work) can be a good thing.
But the biggest problem I think with not being in the room as an interviewer is that you lose the opportunity to hint and direct the candidate away from unproductive solutions or use of time. There are people who won’t ask questions about things that are ambiguous or they’ll misinterpret the program and that shouldn’t be a deal breaker.
Usually it only takes a very subtle nudge to get things back on track, otherwise you wind up getting a solution that’s not at all what you’re looking for (and more importantly, doesn’t demonstrate the knowledge you’re looking for). Or maybe you wind up with barely a solution because the candidate spent most of their time spinning their wheels. A good portion of the questions I ask during an interview serve this purpose of keeping the focus of the candidate on the right things.
hey, I love this idea, but we tried it and we kept getting candidates who managed to BS their way onsite and then waste our time ultimately.
it just didn’t work. I really want it to work because I hated live coding too but it just didn’t.
you can make live coding interviews that aren’t actually difficult questions and are more about showing that you can think and write the most basic of code. that’s what we do now.