The DevOps hiring process should be audition-based
How do we ensure competence? Certification is now a big business, but the way we test a 6 year old should differ from the way we test a doctoral candidate or DevOps engineer.
Back when typists were in demand, companies administered tests for secretarial jobs. If you could prove you had the skill to type X number of words per minute without error, you got the job; hence the term, skilled labor. The testing process is similarly straightforward for bus drivers or professional musicians. You take a driving test. You audition for the orchestra.
This practice doesn't quite work for every occupation. How does a hospital determine the competence of a prospective brain surgeon? Give her a patient and see how it all turns out? Of course not.
For some types of work, a single testing session won't ensure competence. That's why employers might require a college degree or certifications from trusted testing authorities.
When it comes to IT and DevOps hiring, we're relying almost solely on certification. A master's degree in computer science will get your résumé on the desk of a hiring manager at Big Company Inc. But, if you're applying for a DevOps position, being a Certified AWS DevOps Engineer Professional will actually get your résumé read.
Certifications have benefits and drawbacks
Certifications are a big deal, and certification preparation is becoming a big business. These companies provide the instruction and mock testing experience that'll help you pass a given certification exam.
Training companies are popping up all over the place. And, as more companies appear on the horizon, the cost of training -- particularly online training -- continues to drop. You can take a 32-hour DevOps engineer course online for $79. This is the good news.
You can take a 32-hour DevOps engineer course online for $79. This is the good news.
The bad news is that with so much emphasis on test preparation, we're seeing some questionable testing practices. Don't get me wrong. I think testing is valuable. I want the neurosurgeon who's poking around my brain to have passed every relevant medical exam known to man. I want my accountant to be a certified public accountant. The tests these professionals take are issued by established authorities.
However, as we see hiring-by-test-score become more prominent in IT human resources circles, the number of poorly designed tests making their way into the qualification and DevOps hiring process is growing.
Test design is an art
Technical test design is actually a profession in itself. It's a whole lot more than posing a question and asking the test taker to select the right answer in a multiple-choice format. Not only should you be an expert, but you should be clear about the behavior your tests are designed to measure.
Remember, cognition is hierarchical -- think Bloom's taxonomy. No single testing instrument applies to all levels. For example, the test you give to find out if a 6 year old can spell the word cat is much different than the test administered to a graduate student attempting to qualify for a doctorate in the history of feline symbolism in ancient Egypt. The thinking is different. The testing instruments must be different. A multiple-choice question is an adequate testing method for spelling. To test the graduate student? Not so much.
Yet multiple-choice questions dominate IT testing. Take a look at these sample test questions for the AWS Certified DevOps Engineer - Professional exam. Rarely do you see the essay format. Why? Because it's easier to use automation to evaluate multiple-choice questions. The candidate takes the test online, a machine does the grading. The test score become another blip in the automated hiring process -- easy peasy.
What does this mean for DevOps hiring?
Does this mean that companies should do away with technical testing as part of the hiring process? Of course not, nor should the New York Philharmonic should do away with the audition process when hiring an oboe player. But, the Philharmonic is smart enough to realize that you can't identify a competent oboist by having the musician take a multiple-choice test. Talent comes out by playing the music.
When it comes to testing for DevOps competence, it's not so much the testing part of the certification and hiring process that's bad, but that we're using inappropriate testing instruments for the need at hand -- in this case, DevOps hiring.
Sadly, most testing in the technical domain is no more than a spelling test. Only instead of spelling cat, the test taker describes the details of something as special and fast-changing as the Kubernetes configuration specification. Does such testing create a workforce committed to technical excellence? Depends on how you define excellence, I guess. If excellence is about getting the right answer, then the current trends will suffice. If excellence is about creating innovative systems that meet the needs of customers and then some, then forego the spelling test and emphasize the audition. To quote one of my favorite educational philosophers (and Aerosmith guitarist) Joe Perry, "Let the music do the talking."
