Posted by Per, October 2010
Applying Atlassian style recruiting to your "regular" small organisation.
Hiring Great Developers, Part 2: The Pre-Screening
I don't care about your pretty reference letters, your top notch diplomas, your glowing endorsements on LinkedIn. Everyone is a rockstar in their resume! I prefer to send you a couple of simple coding tasks and code review exercises, and ask your opinion on one or two topics. This gives me much better insight into whether it is worth your time and mine to proceed with the recruiting process.
We were going to hire about 20 developers at Atlassian in 2008 while the Global Financial Crisis was still on. We were going to advertise this heavily, so we expected hundreds of applications within days. One of our recruiting assets was speed - making competitors look lame by being fast and efficient, and the new campaign was not to violate the "two weeks max" rule despite the expected flood of applications.
Also, we wanted to keep developing our products, and not send half our development- and management-staff into recruiting marathons day in day out.
So we added an extra step at the start of our recruiting process: Instead of calling applicants for a technical phone screen right away, we started sending developers simple tasks, and we'd have our developers review the answers in bulk-mode, 10 in a row for example. Clearly, we were not the people to invent this. But I am still surprised at how few companies do this.
During that campaign, thousands of applications arrived within 2 months. 30% of those didn't even bother submitting answers to the screening questions we sent. Out of those who did, we could easily eliminate 50% by looking at the answers for about 5 to 10 minutes. Since this was bulk-work, it didn't affect the developers too badly. You'd do it in the morning, curse that morning, but then move on to more fun tasks for the rest of the day.
And since reviewing coding answers is much more developer-y than looking for awesomeness or red flags in resumes, the burden on the HR team was very small during this part of the process: Two to three developers would do 20 to 40 screenings per week each, and provide 3 to 4 lines of feedback. HR could keep focussing on the real HR work, like arraging the following interviews or dealing with visa complications.
Now, obviously, people can cheat, ask their friends for help or even try their luck on public coding forums on the internet. We saw a couple of those. But: People who do this usually don't have smart friends either. Also, we made sure that there were always some low-hanging fruit to achieve. Even a complete novice should be able to find at least two problems in a code review. And then not even consider asking friends.
There will have been a couple of cheaters, but we rarely received great answers from weak candidates. Also, screening is just to reduce the field and to weed out weak applications, not to spot great applicants. And just to be sure, we disregarded the results from the screening anyway, and based our hiring decisions only on the three onsite-interviews.
Summing it up, the simple step of written pre-screening reduced a field of 1000 candidates to about 350. This is the sharpest decrease in the process, and the most efficient one too. 350 is still a lot, and the process is not for free, but the relative cost is small. Look at this chart to get a better impression.
(These numbers are averaged and changed
over time, but you get it)
Although you could easily weed the same amount of candidates by just dismissing applicants based on their resume, nothing comes close when you want to do a good job based on somewhat objective criteria.
All the developers we hired were positive about the pre-screening, understanding why we had to do this. Some even enjoyed it and saw it as yet another reason to work at Atlassian.
Trying this at your own company.
You might think this only makes sense for an attractive employer like Atlassian, and that your "average" organisation can't afford to send screening questions, because applicants would be put off? On the contrary! A good candidate will actually be glad to get these screening questions. He or she will know that you take hiring seriously, and be delighted to get a chance to work at an organisation that tries to hire good employees only. After all, every good developer just hates working with underperformers.
In fact, I am using pre-screening even for my mini-startup (360 degree feedback and performance appraisal software). I don't have an office, nor regular employees, nor a selling product yet. My website clearly lists myself as the sole fulltime-employee. What could be worse for an aspiring developer! And still, I am getting applications by some really smart students (see previous post about writing good job ads), and yes, they do answer my questions and actually enjoy it (within reason).
The key is making the screening questions interesting. I am not asking anyone to solve a hard mathematical problem, or to code for me for free. I am asking aspiring developers to write some unit tests. And since we're working with Google App Engine, I am asking them to summarise the key point of the first 20 minutes of an introductory Google I/O video presentation that's actually fun to watch, and which teaches you a lot about Non-SQL databases. I am asking them to learn enough about a web framework to be able to write a very basic program. I am asking them to review some really dumb code. And since I don't need code-monkeys but employees with opinions, I am asking them for their opinion about my 360 degree protoype: What would you improve? All of these questions are helping them to learn new stuff. They benefit even if I don't hire them.
Have a look over here: We're currently even evaluating pre-screening for our online-marketing internship. If that works out as well, I'll probably switch careers and become a marketing recruiting advisor! :-)
Imagine a world in which every company sends screening questions. Naturally, a job-seeker wouldn't be able to apply at every company since it would take a lot of time. But they could pick those companies that send the most interesting and appropriate questions. Wouldn't that be great for both sides?
Finetuning. Bits and pieces.
Avoiding accidental elimination At Atlassian, we were very worried about accidentally eliminating good candidates from the process. So it was vital that pre-screeners had similar standards. We'd ramp up pre-screeners in pairs, tell them to first look at previous screening results, and to cross-check a few of their first assessments between them. We'd rather promote one candidate too many to the next stage, than one too few. Nonetheless, there was rarely any dispute between reviewers. If a candidate's results were poor, they were ususally so poor that every reviewer instantly agreed to not promote them to the next stage.
Customising the question-set In my own start-up, I am even fine-tuning who gets what question. Not everyone is sent the same questions. Although the work overlaps, I am advertising two roles: One for back-end-only developers, one for those who like back- and front-end technology. Also, I am asking applicants to tell me about their 5 to 10 strengths instead of sending a lengthy CV. The more promising an initial application looks, the more I spend on picking the right questions for the candidate. And sometimes I even let the applicant decide what questions they want to answer, for example if they claim to be strong both in back-end and front-end technology.
With some applications you can immediately tell that they will never reply. But there are some that sound terrific, and I don't hear back from them either. So what! If they can't be bothered to invest a few hours learning about cool new technology and solving trivial coding tasks, I don't even want to talk to them.
Question toughness Our goal at Atlassian was to enable a decent developer to answer the questions in about 2 hours. (We tested this time-requirement on internal developers, just to be sure!). We simply assumed that top developers would probably not be out of job, but working their heads off, possibly have a partner and kids, and not be willing to invest that much time. I hear some consultancies expect candidates to invest 2 days to solve screening tasks. Uh, I believe you'd have to be very desperate for a job to invest 2 days for a regular developer position... Still, my own screening questions for Small Improvements are slightly harder than the ones at Atlassian, since I am currently "just" looking for students. I expect people who want a job to invest around 4 hours, knowing that students won't have quite such a busy life as full-time employees. And yes, people do make it. My "solution received per applications" quota is a lot smaller, but also I have to do less weeding in the following stages. Plus, the developers who I actually interview are smarter, making the interview more enjoyable too.
In fact, I only got very few weak answer so far, and I even have the time to point out the weaknesses in the applicant's solutions, occasionally suggesting they try applying again in a couple of months, after getting more hands-on experience.
Too few applications to justify this? Do you think you could skip the screening because you have so few applications anyway? Well, first of all I believe that pre-screening helps you tell persistent people from slacking ones, no matter how few applications you got. But your main problem is: Why do you get so few applications in the first place!? You should get swamped with applications! If you're not, then you're either a terrible organisation to work for, or (more likely) you're doing a terrible job advertising your jobs. Maybe your job ad doesn't stand out. Maybe you're the wrong have posted it in the wrong spot? Maybe your ad makes you sound boring? Whatever. You should get so many applications that you want to think about screening. Same shameless plug as before: check the other post about writing good job ads.
Anti-discrimination There's another nice side effect of pre-screening tasks. There will be far less people who can complain about being discriminated against. Often, you're able to tell right from the cover letter that an applicant is not good enough for your position. And yet, you might feel compelled to give them a chance to avoid bad press or even being sued. By treating everyone the same and applying same standards, you'll hear back from far less weak applicants, and you'll also be able to point out why exactly you're turning someone down who sent a weak answer.
It also works the other way round. After rejecting 99 weak applicants with a similar background, what's the chance that you'll spot the gem in canidate 100? We're only human, and tend to see patterns even when there are none. But if you send the screening questions to everyone (it doesn't take more than a few seconds), and only hear back from maybe 20 of those weak candidates, your level of frustration will not be as high to begin with. And if candidate #100 does send great answers, you will spot it right away. And then you'd be really dumb (and probably racist, sexist or age-ist too) if you did not promote this applicant to round 2.
Addressing some feedback
A couple of readers had some concerns and feedback, so I'll try to address the most common concerns.To be perfectly clear: To spot a great developer, you need to invest a significant amount of time into the interaction. Coding test, technical interviews with several interviewers, social skills interview, HR checks, visa complications, etc. The time investment is totally worth it, hiring is the most important process in your entire organisation. Even your nicest agile dev process will fail if all you have is mediocre developers!
But you can't spend 10h+ on every applicant if you get hundreds or thousands of applications. Unless you are filthy rich, like Google or Facebook maybe. This post is just about reducing the field in a way which I believe is more fair and less error-prone than others. And my following posts will deal with the actual interviews.
There's been a few concerns raised on Ycombinator that I want to answer. The main points so far:
Commodity!? Absolutely not. Check out the perks Atlassian offers to its employees. There are few companies in the world that do it better. Yeah, I am not interested in really weak applicants who, despite great-sounding diplomas, certificates and x years in the industry can't write a simple algorithm and some tests. But those who can will get an excellent treatment in the next stages of the interview process.
It will be part of the next installment, but yeah, each good applicant would have met at least 4 developers and two managers and two people from HR by the time they signed their contract. Our time investment in good applicants was huge. But it's only worth it for applicants who can write and review at least some basic code.
I don't think that 2 hours is a huge time investment. In my specific case, I expect a bit more than that, but then again I tell people specifically to cut the application itself short. No need to send me a well-crafted resume/cv, save the time for the screening.
I don't know how Facebook does it, and Google did indeed not have pre-screening back then. But you are comparing BILLION dollar companies with a solid mid-range shop. Atlassian had about 200 employees last year when we did this. Google is hundred times bigger, Facebook 10 times. And that's just the people. Just think of all the money they can throw at a problem. We could not, we had to be very careful. Also, back then Google was so slow-moving that Atlassian actually hired new staff who had gotten tired waiting in the Google queue (but were still in it!). I guess Google is faster these days. But don't assume they don't grill you properly. They just have more resources to do it all on-site.
Also, on the supposed hot-ness of Google recruiting: When Google called me to do a reference-check for a former coworker of mine who was applying with them -- it was pretty off-the-mill as well, nothing exciting nor unexpected that Google asked me about him. 10 questions of yawn. (Yeah, he got the job!).
Maybe I should have made clearer just how weak most applicants are. Maybe you're smart, but once you have seen the piles and piles of applications, and you call the applicants and most barely know how to write a method, let alone explain multi-threading, OO or unit-testing, despite years of full-time experience with Java, you start thinking differently. It is a huge time sink to even talk to people for 5 minutes on the phone if many of them are just making stuff up in their resumes.
(Someone has to schedule those calls as well, after all). So yes, I am not kidding, I do value people's time a lot, but if you want a job, I'd rather you spend your valued time on code, than on fine-tuning and padding your CV. Even if you applied at 10 companies, and each asked for 2h of your time, and you picked 5 that seemed worthwhile. Seriously, 10 hours is a Sunday or so, not more. It should be worth it. We're talking about working at a great place. It's your career!
Am I missing something here?
Yes. Screening is just to reduce the field. Since people can cheat, we will not take into account brilliant answers from the screening. Only brilliant answers during the coding test and in the technical interview matter. You can't get hired for brilliant screening answers, when your coding on site was average.
Back to part 1: Tuning your company, writing the ad, placing it
l
