Job scraping a Catch-22 for expanding, muddying candidate pool
Job boards often scrape postings from other sites, enabling access to a wider range of applicants. It can also return false positives for titles that cross over to different industries.
Is more always better? That's the question many recruiters ask themselves when they wade into a job board's search results.
Many boards, pressured to remain relevant as job seekers embrace new channels, such as social media and messaging, are strengthening their services by creating a deeper pool of candidate profiles. In many cases, they do that by job scraping or spidering, meaning they use bots to visit other websites and retrieve information for their own use.
That means a subtle, but important, dynamic is at play when recruiters consider the size of a job board's candidate pool: Not all of its profiles sit on the board itself. Several boards scrape from other sites, including both public job listings and corporate career sites. As a result, when boards offer access to, say, 10 million profiles, only a portion may actually be direct posts.
Job scraping is standard in the industry, said Jeff Dickey-Chasins, principal of consulting firm JobBoardDoctor LLC. "I'd say at least 60% to 80% of the job boards out there are scraping at some level," he said.
Job scraping speeds, and slows, the search process
Standard or not, job scraping can be a double-edged sword. While the practice provides recruiters with a simpler way to review more resumes in one place, the wider search universe often generates a number of false positives. When that happens, users must wade through a greater number of search results to identify candidates who match their needs.
That can soften a job site's marketing message, especially for niche boards that position themselves as providers of targeted sets of candidates. For example, when a construction industry job board scrapes resumes from general sites, searching the term "project manager" may return resumes of IT project managers, pharmaceutical project managers and manufacturing project managers. Few of those are likely to fit the construction job's requirements.
"I think this is definitely the case a lot of the time, especially when there are lots of job titles that can cross over," said Lance Haun, California practice director for the Starr Conspiracy, a communications and strategy company focused on human capital management businesses.
The same thing holds true for candidates, Haun pointed out. A biologist who includes "laboratory" in their search may find themselves with several results that have nothing to do with their specialty. "Laboratories do all kinds of things, not just biological sciences," he said.
In addition, job seekers presented with a wide range of jobs are often hesitant to narrow their search for fear of missing opportunities.
"It's something that's frustrating on both sides," Haun said.
Seeking a competitive edge
Today's landscape is a tough one for job boards, according to industry observers. With the labor market tight, candidates are less inclined to actively look for work online. Many resent the recruiters who spam them, regardless of whether or not they're qualified for a particular position. On top of that, job boards are responsible for only 15% of all hires made, according to the recruiting software provider Jobvite.
Yet, these boards represent a good-sized business. In 2017, their revenue reached nearly $14 billion, Staffing Industry Analysts reported -- a 14% increase from the previous year. While that's a lot of money, keep it in perspective: The U.S. staffing and recruiting market is worth $150 billion overall, according to Statista. Job boards, obviously, are hungry to win a bigger piece of the pie.
In order to entice employers to post jobs -- and generate the lion's share of their revenue -- job boards must offer a deep pool of candidates coupled with powerful search tools. Last year, several sites, including the wide-ranging Indeed and the niche technology board Dice, unveiled AI-based search tools that help recruiters zero in on candidates most likely to meet the requirements of a particular requisition.
Editor's note: Mark Feffer was previously employed by Dice.
Job scraping in the modern age
Will Kelly, a longtime recruiter who now consults with organizations on hiring best practices, said he believes job scraping can be valuable "when you're looking for any piece of information possible, whether it's contact information or where someone currently works."
That kind of data isn't readily available, according to recruiters, because the most promising candidates are often passive, meaning they're not looking for a new position and haven't posted their resume anywhere. Because of that, experienced recruiters recognize that finding the best match often requires detective work.
"We're all investigators now," Kelly said. "Sometimes, you just need one little piece of information" to connect with a particularly promising candidate. "To me, job scraping is doing anything you can to get any and all information you might need on somebody."
Job boards and third-party developers have noticed more recruiters are thinking that way. They're creating advanced tools to help them more easily sift through job scraping results.
To me, job scraping is doing anything you can to get any and all information you might need on somebody.
Will KellyLongtime recruiter and consultant
For example, AI-driven systems can review all of the profiles uncovered for a particular position and determine their median tenure, said Joseph Hanna, founder and CEO of Engage Talent, an AI platform that helps companies identify and target candidates. If that tenure is seven years, the system can spotlight candidates with 10 years of experience as prospects who are overdue for a chance to move up.
Some systems go further. They can parse data so job seekers can more easily identify employers whose culture aligns with their beliefs about work. For instance, a system could prioritize organizations known for their training programs and commitment to internal hiring. Those factors could mean a lot to candidates who put their stock in professional growth, Hanna said. On the other side of the table, such systems can identify candidates who won't be attracted to organizations that spend little on training and development.
Such tools aren't limited to the use of job boards, Dickey-Chasins said. "Most midsize to large companies typically have lots and lots of candidates in their applicant tracking systems, but ATS vendors haven't done the best job of allowing them to get in there and use them," he said. "A new breed of tool" not only allows sophisticated searching of internal databases, but conducts job scraping of public databases, as well.
"I think those tools are actually very valuable. Based on what I've heard from employers, they're really useful," Dickey-Chasins said. "But again, you have to go back and say it's totally dependent on the data and the quality of the tool itself, how it parses and lets you evaluate and sort and all that good stuff."
