Testing the Blind Interview Process for Internal Junior QA position

Let's look at how we streamlined the hiring process in our Software dept.!

Favorited Favorite 0

This year we’ve done a bit of hiring in SparkFun’s Software department, and with each new hire we’ve really been focusing on dialing in our interview process. Earlier in the year we hired three developers at various skill levels and experience, so with each round of candidates we wanted to make sure we found the best fit for the department while making the interview experience as pleasant as possible.

We settled into the following format:

  1. Phone Screen: Allows us to learn about the candidate and for us to explain the position and introduce the company to the candidate to see if there’s a match.
  2. Code Repo: Allows us see some samples of the candidate’s code, how much or little is documented within the code, and helps us figure out the candidate’s skill level.
  3. Onsite Interview: General code questions; meet with the team; short, in-person coding exercises; and of course a tour of SparkFun’s headquarters.

Our next challenge was to hire a Junior QA Analyst internally. We decided to go internal-only on this position because:

  • Current employees know how our internal tools work and have domain knowledge already.
  • We wanted to create a gateway for more internal hires into our department, which has historically hired externally.

Like normal, we first defined job requirements and a job description, keeping in mind that we wanted this position to not require any coding or web development experience. At first we only received one applicant, but after a second reminder that the position required zero coding experience, we ended up receiving 12!

Since this was an internal-only posting, we each admitted that we had certain biases towards some candidates. Some of the candidates we’ve worked more with, some we haven’t. After a recent company Lunch & Learn on Inclusion and Diversity brought up blind interviews (see why blind interviews are a thing) as a way to combat our biases, we decided to try out our own blind section for this interview process. These are the core skills that the Junior QA Analyst needed:

  • Clear, concise written communication
  • Interest in learning coding and web development
  • Interest in testing all the things

We also wanted to make the take-home test as work related as possible to give candidates a taste of what the job might be like. We planted several bugs (over 10) on two pages (main home page and a product page) of our public-facing site, and hosted it internally for the candidates to visit and then file “bug reports” via a Google Form. The form asked for a summary, description, steps to reproduce and “what do you think is wrong / how would you fix it?” questions.

Screen shot of the form used to collect applicants responses and explains to provide as much information as possible to identify the bug

We planted a variety of bugs: broken links, fake image URLs, input fields set to readonly, different logos in different browsers, broken JavaScript, common typos/math errors and more. We wanted to see if candidates were checking various screen sizes, browsers and the JavaScript console. We also wanted to get an understanding for their current development skills if they have any, for example: Can they figure out that the link to open the modal is looking for “#shipping-restrictions,” but it is actually “#shipping_restrictions” so nothing happens when you click it? Most importantly, we wanted to see how well a candidate could describe a bug. Figuring out what went wrong was just gravy on top. We also asked the candidates to only spend 1-2 hours on it because we want to respect their time.

Here are a few screenshots of some bugs, since the site was only available internally:

Screen shot showing the SparkFun logo as a telephone instead of a flame.

SparkFun logo changed based on which browser you were using to view the page.


Screen shot showing an Unhanded exception error

“Unhandled Exception” error only shown on small screens, like a phone.


Screen shot showing the What's on your mind form and the code related to it underneath showing the readonly attribute that should not be there

“What’s on your mind?” form showing that the textarea element has a readonly attribute that prevents the user from inserting text into a required field.

First we met with everyone to do an in-person phone screen and go over the take-home test. Then the candidates were given nine days (spanning two weekends) to complete it. Once the Google Form closed, HR took the responses and swapped the candidate’s name for a numbered one, e.g. “Person 1.” We had 10 candidates complete the take-home with a total of 91 reported bugs!

As a team, we went through all the responses and ranked the “quality” of each bug report from one to five. There were some bugs that were invalid or duplicates, so those didn’t get any points. Some candidates found bugs that we didn’t plant and are valid in production, so of course they got points for those. Other baselines that we established:

  1. Not correct or not helpful
  2. Kinda on the right path, but not there
  3. Doing ok
  4. Good - found the bug, documented it well
  5. Great - found the bug, documented it well and had good ideas on how to fix or what was wrong.

We chose our top three based on their total score (sum of their quality scores) and the number of fives they had. Also conveniently there was a noticeable difference between the third- and fourth-ranked applicant, so having three as our final round really made the most sense.

After notifying our top three about the next step, which was a short, in-person interview, we looked at our fourth- and fifth-ranked individuals who had done really well, but just didn’t quite compare to the top three. We realized that this grading system was biased toward those with previous development experience, but we figured that we would be able to get someone more experienced up to speed faster. Next time we open up this type of position, we may want to ensure that previous dev experience doesn’t have as much of an advantage as it did this time.

Since this was an internal-only position, we had the opportunity and felt that it was important to provide feedback to all the candidates. Our QA Engineer met with each one to discuss what they did well, what they should work on and what resources to look into.

What we really liked about making the test blind was that we only looked at what the candidate provided. We didn’t look up their past bug reports or past performances. If this had been an external candidate, we wouldn’t have known those either. We didn’t give someone that we enjoy working with higher scores. We came to a consensus much faster for our top three than we would have if we didn’t have these objective standards. We were also hoping that the candidates enjoyed the experience too, and the feedback has been positive! Our top candidate feels that they earned the new position on merit rather than biases or office politics, and hopes this type of blind exercise shows up in other job openings. Others believe that it helped to put everyone on the same playing field… which was exactly our goal of the exercise!

We’re looking forward to including more exercises like this for each open position when possible. We feel that the anonymous nature of the “blind” test helped us focus on picking the person with the best skills for the job in a timely fashion, and the candidates felt that they were being compared evenly. Win-win!


Comments 8 comments

  • I like that you took the opportunity to provide feedback to even those who did not make it all the way through. I wish that I knew what other companies that I’ve interviewed with that didn’t hire me thought about me so I could strengthen my weaknesses as a person or even for the same company some time down the road.

    • Giving feedback did take me a bit longer than I had anticipated. I wanted to make sure it was valuable to the candidates, but I also was very encouraged with the feedback I got in return from them all. While there was obviously some disappointment that they didn’t get the job, it was good to hear multiple people say they felt they got a fair shake regardless of whether they knew us personally or not.

      As a bit of a follow-up, we’re also planning on internally providing resources and training for those interested in learning more about the Software QA/Continuous Integration process starting later this fall. Exactly how it will look is something we have yet to figure out, but to those looking for a change of pace in their careers, we’re hoping to provide exactly that.

  • Would be a dream to work at Sparkfun, but 8 years from graduation and health issues making it so I never got an engineering job after college kind of puts me out of the running. Its cool to see some of the processes at least

    • I wouldn’t agree that you’re out of the running. Sure - it might for example be difficult to get straight into a Senior Electrical Engineer position if that’s something we’re hiring for, but every engineer’s path at SparkFun has been a different one. There are plenty of examples of employees who have started in shipping or manual assembly, to then move on to different departments and jobs depending on skill and interest. I do believe we’ve had a quite a few Engineers come through Tech Support, for example.

      As for SWIT, we all started off down different paths too. I think there’s only a couple of people (myself included) who actually have Computer Science degrees. I’m not trying to take anything away from having a relevant education - I’m just pointing out that there are many ways to become a skilled and productive employee in a given industry if you’ve got your sights set on it.

  • I certainly wish the new QA person the best of luck! I’m impressed with the idea of a “blind” test.

    I’ll mention that I’ve never officially had “QA” in my title (that may be one of the reasons that I spent 21 years with the same company – we used to joke that the QA desk was “right beside the door” as it seemed that every time there was a RIF, it included anyone with “QA” in their job title). I did occasionally get asked to “test” other people’s code – and those whose code I’d be testing always cringed because they knew I’d find a lot more bugs than most would find. (I was hardest on my own code, and often got compliments from users about how well mine worked.)

    BTW, I like the possibility of having the submitter suggest ways of fixing. A few months ago, I reported a problem with Firefox, and included a suggestion as to the fix based on my decades of experience – although I never heard any feedback, I suspect I might have made a big difference as there was a new version out within about 48 hours with the problem I’d reported fixed (and I didn’t notice any other changes).

    One last point: I hope that you don’t engage in any “age-ism” in your hiring. Someone with lots of experience may know to look for “bugs” in places that a less experienced person might miss, and for development positions may be able to avoid a lot of “dead-ends” and also know how to avoid many of the problems up front. (Remember that one often maligned group that everyone will eventually be a member of is “old”.)

    • One last point: I hope that you don’t engage in any “age-ism” in your hiring.

      I have to admit that while this selection process would more or less eliminate most “isms,” we’re definitely at a disadvantage in that specific area with internal hires since SparkFun has a large number of younger employees. On top of that, since this was a junior level position, I don’t think there was much interest from the older segment of the company to apply.

      That being said, SWIT also probably has a fairly high median age compared to the rest of the company, though what that exact age is, I haven’t bothered actually finding out. But let’s just say that most of us fall in the range of 30-50 with a few outliers. I honestly do believe we’d happily work next to anybody regardless of where they fall on the age spectrum, though.

      • I certainly didn’t mean to imply that you are engaging in “age-ism”! (Sorry if it sounded like it.) I just wanted you, and other readers, to be aware of the hazard. I can certainly see how with “internal hires” you are restricted to what’s already in the company, and I very much applaud looking within first as it can be of very great benefit both to the company and to the employees. Just be mindful of the hazard when looking for “external hires”.

        I can say that folks who are more experienced sometimes like to undertake something different to have new challenges, recognizing that they very well might have to accept a “junior level” position. And there folks who are dissatisfied with their “higher level” positions, and really would prefer a lower-level position. (Ahem… I recall a certain SparkFuner who did this a while back…)

        • I didn’t read your post that way, I just wanted to clarify that I do believe we’re open to people of all backgrounds as long as they have the skills we’re looking for.

Related Posts

The Move Is On!

On Counting Things

Recent Posts

Why do we collaborate?

Tags


All Tags