Researchers criticize HackerOne over triage, mediation woes

Communication issues

"Unfortunately -- and it pains me to say this because I was their first executive hire many, many years ago -- we've certainly observed a general decline in [HackerOne's] triage services," said Luta Security founder and CEO Katie Moussouris in an interview with TechTarget Editorial.

Moussouris is a pioneer in the vulnerability research and disclosure space, having previously created Microsoft's first bug bounty program. She was also previously HackerOne's chief policy officer from 2014 to 2016.

While HackerOne is continuously adding new features and putting in work, she said, the decline of the platform's communication quality has been "painful to watch" because it's an important part of the cybersecurity ecosystem.

Moussouris said Luta Security, which works with clients to improve and establish bug bounty programs, has observed worsening triage times and judgment accuracy, particularly in the last year or so. Triage refers to the process of a program receiving a vulnerability and judging its validity.

Moussouris also noticed that on a whole, when companies approached Luta Security for help, "it's more often that we get called in, quite frankly, for programs running on HackerOne than for programs running exclusively on Bugcrowd."

Multiple researchers shared their HackerOne communication experiences with TechTarget Editorial. These stories primarily involve triage and mediation -- the latter being a distinct process that occurs when a researcher contests the judgment of a vulnerability submission. This generally happens when the program determines that a bug is a lower severity than the researcher believes, is worth less money than the researcher believes, or is not within the program's scope.

"Historically, mediation with HackerOne has been worthless," said bug bounty researcher Tommy DeVoss.

DeVoss, best known by his handle "dawgyg," is one of the highest-earning researchers to ever bug hunt with HackerOne. He is also one of the first to cross the $2 million barrier in earnings on the platform. Despite that success, DeVoss said HackerOne's mediation "almost never solves anything" -- even for him.

"I hadn't even used [HackerOne mediation] in probably three or four years, because it never once produced results," he said. "Not even good or bad results -- just results period."

DeVoss said that historically, HackerOne mediators would either not respond to tickets or would resolve it in favor of the program because HackerOne's programs, which are operated on behalf of corporate clients, are HackerOne's customers.

Moussouris said that while this is not unique to HackerOne, it is "kind of misleading" for any platform to call the process mediation because a platform would be unlikely put its foot down against its own customers.

HackerOne makes a 20% commission -- a "Rewards Fee" -- on every bug bounty paid, which Moussouris described as a "perverse incentive" and something that its competitors don't have.

"It makes 20% commission on top of every bounty paid. So it's always trying to drive the prices up, and it's always trying to buy things that other programs might not pay for," she said.

Asked about the 20% Rewards Fee, HackerOne CISO and chief hacking officer Chris Evans told TechTarget Editorial via email that the company changed its fee model.

"HackerOne retired its original 20 percent Reward Fee model six years ago," he said. "The phase out is largely complete. HackerOne customers are now charged via a SaaS subscription that includes all HackerOne services they pay for."

As of publication, a 20% fee is still mentioned on HackerOne's terms and conditions page. TechTarget Editorial asked Evans about the current terms page and have not received a response at press time.

UPDATE: Evans said the fee was still referenced on HackerOne's terms and conditions page because it represents a "very small" and "steadily decreasing" fraction of customers, and that it hasn't been offered to new customers in years.

In addition to seven years of hacking on HackerOne, DeVoss runs the Bugcrowd program for engagement platform vendor Braze. He said that in his experience, Bugcrowd will work harder for hackers in mediation and send swag to top hackers each quarter. In comparison, he said, "I honestly don't remember a time even for me that HackerOne actually went to a program and got an issue fixed for me in a positive way."

A prominent security researcher known by the handle "Sick Codes" had a highly publicized exchange involving HackerOne when he discovered two serious vulnerabilities in John Deere farming equipment last year. The researcher told TechTarget Editorial that like others have said, the biggest problem with HackerOne is communication.

"The most common problem with HackerOne is communication," he said last month. "That's all it is -- people speaking to people. There's no bots and no code involved. It's just people telling other people how their bug is issued, and the HackerOne staff is not good enough at it. Their customer service isn't good. As a business, their entire job is to mediate vulnerabilities, and they aren't great at it."

Sick Codes also echoed the sentiment that while Bugcrowd isn't perfect, it "listens to mistakes really quickly and fixes them."

Researcher accounts

Evans said that out of the total "valid mediation requests" sent by hackers to HackerOne this year so far, the platform ruled in favor of hackers 74% of the time. Moreover, he said 2021 maintained the same rate. Evans linked to HackerOne's mediation page to specify what a valid mediation request is.

"Every mediation case is unique and handled on an individual basis," Evans said. "Success in a mediation is defined as getting to an outcome where industry standards are adhered to -- for example, honoring policy and bounty table commitments; assigning correct severity and impact; and paying for work done within a reasonable time frame. We will be publishing more industry standards and best practices over the next months, so customers and hackers alike can understand expectations in the ecosystem."

A number of researchers shared their own mediation stories with TechTarget Editorial.

John Jackson, a researcher as well as founder of the Sakura Samurai hacking group, published a Twitter thread last October describing how, after submitting a vulnerability to Ford and seeing it fixed, the auto manufacturer's program ignored repeated requests for disclosing the bug to the public. Jackson said that HackerOne then violated its own disclosure policy and banned him from Ford's program.

In celebration of Hackerone violating their own 180-day disclosure policy and banning me from Ford, here's our writeup and proofs. Don't worry, it's been patched for over half a year. Hackerone mediation ghosted me after saying that Ford didn't "seem" like a bad company. pic.twitter.com/V0VxG6mFVO

— John Jackson (@johnjhacking) October 1, 2021

"If HackerOne sides with the researcher, they could potentially lose a client and a ton of money since HackerOne charges programs 'x' amount of money yearly," he told TechTarget Editorial. "It's easier for HackerOne to wipe their hands clean and say it's up to the program to decide the final action in disputes. That way they don't risk losing money. In short, mediation isn't a neutral way to help hackers and programs resolve issues. Mediation is HackerOne's best attempt at trying to dissuade or diffuse the anger of a hacker."

Another member of Sakura Samurai, Patrick Martin, described negative experiences he had working with BMW Group's program on HackerOne.

Martin described a situation where he submitted a plain text credential disclosure flaw that did not see a response or reward for 10 months despite the issue being silently fixed. In another case, Martin described an exploitable SQL injection flaw that HackerOne triage representatives allegedly resisted paying out for six months.

Martin shared screenshots of both bug submissions with TechTarget Editorial.

Syed Ali Zain Naqvi, a researcher as well as security engineer at software development firm TechnoGenics, described a similar experience from earlier this year where he felt the triage rep didn't properly understand the bug he was submitting, which was an account verification bypass vulnerability.

The triage representative labeled it "informative," he said -- a designation saved for "useful information but doesn't warrant immediate action or a fix," according to HackerOne's website. Naqvi said it was this interaction that caused him to leave HackerOne as a whole.Not all interactions with HackerOne have been negative. Some researchers declined to comment on the record for this story, citing positive experiences with the platform. A number of posts on websites like Twitter and Reddit also show HackerOne being readily praised and recommended as a bug hunting platform.

However, stories from researchers about HackerOne taking months to respond to tickets and being unhelpful in both triage and mediation are not uncommon.

Atredis Partners research consulting director Justin Kennedy published a Twitter thread on Oct. 19 describing a situation where he and a friend submitted 10 vulnerabilities, which they say were critical and high severity bugs, to a HackerOne program. He said it had been two months since the bugs were submitted and more than one month since the duo initiated a still-unresolved mediation "when the program tried to screw us out of those bugs."

In the replies, several researchers shared their own negative experiences communicating with the platform, similarly citing "useless mediation" and long wait times. Evans responded to the Twitter thread and asked Kennedy to provide additional information.

It has been 2 months since a friend and I submitted ~10 crits and highs to a program on @Hacker0x01 (100+ hours spent on the program) and over a month since we requested mediation when the program tried to screw us out of those bugs. And it's still not resolved. Ridiculous.

— Justin Kennedy (@jstnkndy) October 19, 2022

In a direct message, Kennedy declined to comment further beyond issues cited in his thread but said TechTarget Editorial was welcome to reference his public tweets.

Potential signs of long-term improvement

On Oct. 17, HackerOne published a blog post detailing insights it gained from HackerOne's Las Vegas hacking conference H1-702. The post described how members of the mediation team sat down with members of the HackerOne community to solicit feedback while also signaling ongoing improvements to the mediation process.

"There is work to be done regarding response times to mediation requests," the post read. "We have since increased our headcount and hope to continue in 2023. We are also looking into ways to make backend mediation processes more efficient with the help of the HackerOne Product Team."

The post also described the need to both spread awareness about how mediation works and provide greater transparency into its success rate as well as the most effective use cases for mediation. Similarly, HackerOne's Evans said that, "when mediation succeeds, no-one tends to hear about it."

"It's on us to tell some of the unheard stories where a hacker’s report was not assessed correctly, and we stepped in and fixed it," he said.

In a second blog post, HackerOne described its "Make It Right" fund. The fund was established for when HackerOne believes a researcher's bug submission has been mishandled and believes said researcher should be awarded.

HackerOne's Evans told TechTarget Editorial that the platform created the fund in 2020 as an "internal discretionary fund" and recently announced its existence "as part of our efforts to increase transparency."

He said that this year to date, HackerOne has awarded researchers $38,650 through the fund, adding that he has personally been involved in some cases to ensure the right outcome is reached."I'm pleased that we have the option of the Make It Right fund, but the goal is to work to bring all customer programs in line with industry standards," Evans said. "I’m working on defining these, and requiring programs to adhere to them. As we get closer to this goal, we will not need to Make It Right as often."

HackerOne triage representatives are also apparently changing their usernames to help researchers discern when they're talking to a HackerOne employee and when they're communicating with a hacker contracted to assist in the triage process. Sick Codes commended the change and called it "huge" for transparency.

DeVoss, who is currently in the process of building a bug bounty researcher advocacy group, participated in one of the H1-702 mediation meetings. Though he didn't share specifics, he said he felt HackerOne was starting to make strides in the right direction.

"I did spend quite a bit of time with the mediation team in Las Vegas. And they are making some pretty big strides in trying to rectify because they, from my perception, recognize that when pretty much every single person who deals with mediation says it's broken, they can't all be wrong," he said. "They are hiring new people and trying to move forward."

DeVoss said that he loves and is grateful to HackerOne for everything the company has done for his personal and professional life. But a pattern of HackerOne promising improvements without following through has led to him more readily speak out about these issues.

He said he's building an advocacy group because currently, there are countless researchers not being advocated for who don't feel like they're being seen or heard. DeVoss wants his group to be supported by himself and other prominent voices in the researcher community -- while he may have the ability to directly contact someone from HackerOne if he's having issues, others don't.

"I don't generally have a lot of problems that don't get solved," he said. "And I mean, as much as it sucks, it's just because of who I am. They help me out when I have problems, or they help me get resolutions. It's not always the resolution that I want, but I don't feel generally that I'm being ignored the way that other people do. I want to do this so I can help those people."

Asked if he had any additional thoughts for this story, Evans told TechTarget Editorial that HackerOne values both its hackers and its customers.

"We recognize that improving mediation is an ongoing journey," he said. "I'm a hacker myself, and the reason I’ve been hired as Chief Hacking Officer is to ensure hackers are treated fairly. I spend much of my time thinking of ways to do this that are scalable. I’m excited to share more notes and updates over the coming weeks and months."

Alexander Culafi is a writer, journalist and podcaster based in Boston.