Data-driven decision-making case study: Indeed

Time to hire index

Indeed's goal is to help employers find the right employees -- and vice versa -- quickly and efficiently, so its products must enable that task. But Indeed employees did not have full visibility into which of its products supported that strategic objective, said Sowmya Parthan, Indeed's director of data and analytics. They used experience and anecdotal evidence to determine which product improvements they should make to boost hiring efficiency.

Indeed needed more insights into its products to drive decisions about which features would improve hiring efficiency, and to understand how to prioritize work on new tools as well as improvements on existing ones. The company had to identify, collect and analyze data that could provide insights into the employer hiring experience.

"There was a concerted effort around doing better measurements, and that's how the time to hire index came along," Parthan said. "When an employer comes to us to post a job, they have a process to screen, interview and hire. From the time the job is posted to hiring, that's 'time to hire,'" she said.

The time to hire index contains temporal data on each hiring journey. Indeed had long tracked the times it took for employers to reach the various milestones along completed hiring journeys. It could analyze data by job segment, employer attribute and benchmark events, such as "job creation" and "first apply." Time to hire data enables Indeed's workers to make faster and better decisions as they devise, prioritize and work on projects that make the hiring process more effective and efficient.

Indeed's leaders recognized that they needed more granular insights from the data about which of its products they could use to improve movement from one milestone to the next. So, teams identified the Indeed products that supported the hiring journey and studied the mature data sets those products produced. Now Indeed uses that data to find the right opportunities to improve time to hire. For example, Indeed workers can use the data to test whether a product improvement or a new feature moves the needle.

"We can do that kind of test more easily now because we have all this data, so we know which improvements to chase and how to prioritize them," Parthan said.

Cloud spend as a percentage of revenue

When Indeed moved its production services from its on-premises data center to the cloud, executives recognized that they needed to determine how best to assess its cloud spending, said Jeff Davis, vice president of engineering.

"We hadn't had experience in managing cloud resources at scale, and we wanted to know if our approach was healthy and sustainable," Davis said.

Indeed wasn't concerned with whether its cloud spending was less than its on-premises costs, he said. The company switched to the cloud for improved security, resiliency and application development capabilities. It needed a metric that looked beyond the dollar amount it paid to its cloud providers; it required a measure that would include all the IT and business-related considerations.

To determine accurately whether the company's cloud strategy was on the right track, software engineer leaders and IT leadership proposed a data-driven metric called "cloud spend as a percentage of revenue," which they call CSPR internally.

"CSPR was developed to determine if we are doing the right thing for the business and to let us answer that question continually over time," Davis said.

Cloud spend as a percentage of revenue isn't just a number to report; it helps Indeed determine how it will spend its resources. If the number gets too high, it signals to the company that it should focus its engineering efforts on cost optimization to bring CSPR into the target range. If it gets low, that indicates that Indeed should consider new cloud investments and innovation to support business growth or risk reduction. If it falls within the target range, the company is spot on with its allocations.

"It's powerful because it lets us know how the business is doing, how much we're spending on this infrastructure and how that compares with our own historical data and our peers," Davis said. "It helps us know how much we should invest in optimization, or whether we should spend on innovation."

The engineering team applied its standard approach to develop cloud spend as a percentage of revenue, starting with a proof of concept to prove value and then iterating on it.

The team collected historical data, mainly on hosting costs and top-line revenue. Staff from the finance and procurement departments aided in the effort, supplying additional data for past on-premises spending costs to help provide a baseline for analysis.

Project leaders also collected cloud spending data from peers to help Indeed benchmark against companies in its industry. This was one of the most challenging parts of creating CSPR, Davis said. Most of the data came from revenue statements issued by publicly held companies or professional networks. Cloud spend as a percentage of revenue also pulls data from cloud providers using APIs.

The team worked with a spreadsheet to start but added automation and additional tools to reduce the manual workload. CSPR now uses Indeed's analytics platform, Imhotep, which can hold data and support ad hoc and automated reporting needs.

Creating CSPR was not a daunting undertaking, which shows that "making data-driven decisions doesn't have to be overly complicated," Davis said.

Targeted surveys

The Net Promoter Score (NPS) is one of the most widely used metrics within organizations to measure the effectiveness of their customer experience programs. Indeed executives found NPS to be a poor measure for understanding where and how Indeed could improve specific components of the experience it provides for its users, said Adam Hagerman, UX research director.

NPS represents the entire user experience, so it has value in that it shows senior executives an overall picture. However, Indeed needed a disaggregated measure to understand what to improve; a high NPS doesn't provide information about what part of the total experience drives that high score.

"We needed information about how users are finding value in our different products," Hagerman said.

Now Indeed uses an internal system called ASK'EM instead of NPS. The ASK'EM program uses findings from targeted surveys, which employers and job seekers take at key points in the hiring journey, to garner customer sentiments. ASK'EM uses the data to decide where and what to improve. For example, employers posting job requirements might receive a survey that asks, "How satisfied are you with your ability to define the requirements for your job?"

"We wanted to see an increase in the ASK'EM score for an experience's purpose," Hagerman said. "We'd want to see that each successive improvement we ship helps to increase that rating or at least doesn't cause it to go down."

Hagerman detailed the steps Indeed followed to adjust from using NPS to using data from its targeted surveys. First, Indeed employees identified key points in the user experience that have meaningful relationships to positive hiring outcomes. They also identified where to place surveys and what questions to ask different types of users. For employers, common survey points include after they post a job, after they review a certain number of resumes and after they close the job.

Hagerman's team then developed a proof of concept and scaled it up.

"We did a lot of research to define what the measurement should be, understand what [Indeed users] were trying to do and who were these people so we can categorize them," he said.

"We had to segment people properly, so we had to do segmentation research. And we had to articulate the desired path for each category. So, we had that body of research to get first, and then we used that to understand what the correct measurement points should be," Hagerman said.

Indeed deploys surveys across its suite of experiences including job posting and candidate management workflows, job seekers' homepages, analytics pages and resume search. The survey data enables executives to identify accurately where and what to improve, and how to allocate resources and prioritize decisions.

"We're collecting data, analyzing the data, to show where we need to focus our user experience work," Hagerman said. "NPS would never tell us where to reallocate resources to improve user experience, but this program does."

Mary K. Pratt is an award-winning freelance journalist with a focus on covering enterprise IT and cybersecurity management.