Border crossings. Police checkpoints. Security cages. Secret safes. These are just some of the hurdles certificate authorities face as they strive to maintain security during COVID-19.
Certificate authority oversight was complex and arduous prior to COVID-19, but social distancing requirements, quarantines and shelter-in-place orders have made the in-person meetings required to secure and verify digital certificates enormously difficult.
For example, the major web browsers require certificate authorities (CAs) to undergo annual audits, which are conducted by third-party firms on site. More importantly, CAs conduct key ceremonies, which are often used to generate public and private keys for digital certificates, or even to revoke or destroy compromised certificates.
Most key ceremonies not only require auditors but notaries, legal representatives, official witnesses and key ceremony "masters" and managers from the CA itself. The ceremonies are conducted only by authorized personnel in secure facilities; access to these facilities often require several layers of authentication, including smart cards and biometric scans. A missed audit or a delayed key generation ceremony could have devastating consequences not just for the CA in question, but for every HTTPS website, code-signed application and authenticated digital document that relies on the CA's certificates.
"We have audits that have to happen, and we have key material that has to be kept safely in data centers, which requires people to physically travel to them," said Nick France, CTO of SSL at Sectigo (formerly Comodo).
Under the best of circumstances, such events involve meticulous planning and painstaking processes. The pandemic has made already-complex auditing and key management operations even more challenging at a time when abuse of certificates for phishing attacks has increased, according to Arvid Vermote, CISO at U.K.-based GlobalSign.
Some of the largest CAs in the world said they are handling key management events under the current circumstances. But it's unclear how long that may last, especially if the situation worsens, and what that will mean for a cornerstone of web security.
Key ceremony challenges
Most CAs have extensive business continuity and disaster recovery plans to ensure root keys are protected, generated and revoked as needed. But they admit that nothing has quite prepared them for COVID-19.
A few years ago, GlobalSign addressed the risks around the security and availability of root keys, Vermote said. The company has locations in three different continents where it holds active pairs of roots keys and can manage keys in case one or even two locations becomes inaccessible due to natural disaster, a geopolitical disruption or some other kind of cataclysmic event.
"We initially addressed those concerns, and we thought that would be enough to ensure resilience," Vermote said.
But circumstances changed when the global pandemic affected all three of GlobalSign's regions. GlobalSign and other CAs have been designated essential businesses across the vast majority of countries and regions in which they operate, which allows staff members to travel and access data centers for critical key management ceremonies.
Participants gather in a security cage for a DNSSEC key signing ceremony in 2016.
But because those activities require a significant amount of work -- and risk -- during the pandemic, CAs have tried to adapt their practices and schedules. For example, DigiCert, a major CA based in Lehi, Utah, said it's delayed some non-essential key management activities to prioritize the safety of its employees and "promote social distancing to flatten the curve of this pandemic," a company spokesperson said.
DigiCert is in touch with customers to anticipate requests and schedule them accordingly, and does not foresee significant impacts, the spokesperson said.
Some key ceremonies are easier to pull off than others. For example, France said Sectigo regularly performs key signing ceremonies for subordinate or "sub-CAs." "A lot of our partners have a sub-CA that's branded to them from our root certificates. We maintain and run the sub-CA -- it's a white-label exercise, and obviously because their certificates are signed from the roots, we have to go into the data centers and perform the ceremonies."
While key signing ceremonies occur more frequently than, say, a key generation ceremony, they don't require as many participants; auditors are recommended but not required. Luckily, France said, Sectigo had key generation ceremonies late last year and earlier this year prior to the pandemic.
Olaf Kolkman retrieves key material from a hardware security module during a DNSSEC key signing ceremony in 2016.
"We're not too concerned because those key generation ceremonies, at least for us, generate a large number of keys at one time," he said. "But we do have a need to create certificates based on those keys, which does require access to offline signing devices in multi-layered, secured cages within these data centers."
Rather than delay key management ceremonies, some organizations opt for a different approach. The Internet Assigned Number Authority (IANA), which manages the allocation of global IP addresses and DNS root zones, has a DNSSec root signing ceremony scheduled for April 23 in the U.S. Instead of postponing the event, ICANN, which oversees the IANA, decided to sign nine months' worth of signatures in one ceremony instead of conducting one every 90 days for the rest of the year.
But there is a tradeoff, according to Olaf Kolkman, principal for internet technology, policy and advocacy at the Internet Society.
"The reason why you sign keys every quarter is risk mitigation," he said. "If one of the zone signing keys gets compromised, you at least have a limited time of exposure [of 90 days]."
Kolkman, who volunteers at the IANA as a crypto officer and has participated in DNSEC root signing ceremonies in the past, said signing keys for future use extends the window for when those keys could potentially be stolen or exposed. Cybercriminals could use keys for code-signing malware, and nation-state actors could use them for elaborate cyberespionage campaigns -- or, worse, attacks that disrupt large portions of the internet itself.
"You get into movie script-type scenarios," he said.
Then there are there more immediate health concerns that come with holding key ceremonies during the pandemic.
Kim Davies, vice president of IANA services at ICANN, said the organization tries to minimize the number of participants who need to be physically present for a ceremony. Thursday's event will have seven people on site, but past ceremonies, such as a 2016 DNSSEC key rollover, had more than 20.
"We ultimately settled on an approach where we tried to involve everyone as much as possible as they normally would, but in a remote way," he said, adding that ICANN set up secure videoconferencing channels for remote participants.
While the ceremony room in the ICANN data center can hold 30 people comfortably, Davies said, the security cage or safe room is a different story. These areas are where the key material is held in hardware security modules (HSMs), and they're typically 6 feet by 12 feet. The ceremony requires three people in the safe room at one time.
ICANN acted early and procured personal protection equipment (PPE) for staff members in January, according to Davies. "It was certainly a point of discussion whether we wanted full-body suits for the ceremony," he said, "but we ultimately assessed that wasn't necessary."
'The roadshow'
Not all key management ceremonies can be delayed or planned weeks or months in advance. Certificate authorities will occasionally have emergencies where root keys need to be revoked or even destroyed. And in those cases, the ceremonies must go on.
GlobalSign faced this exact situation recently; the company discovered it had a compliance issue that required the destruction of potentially compromised keys, which forced Vermote and other employees to conduct a "roadshow" earlier this month.
"Normally when you're planning for global key management, you think about key generation and revocation of certificates," Vermote said. "But you don't think about key destruction."
For key generation or signing a certificate revocation list (CRL), you only need one key management location. But for key destruction, guidelines state you need to destroy all the copies of the keys in your possession, which could require traveling to multiple data centers.
"The challenge we had in the last month was we had to destroy all copies of these keys in all of our global locations, even if all of them were under lockdown or semi-lockdown orders. So that was quite interesting," Vermote said.
The process was a challenge, he said, because GlobalSign requires five to six trusted individuals to get access to the root keys in all of the locations. That involves a significant amount of travel and a risk of exposure to the coronavirus. And, Vermote said, those individuals are often some of the more high-profile executives in the company, including himself. If one of them if infected, it disrupts the business continuity plan for key management and also puts other GlobalSign employees at risk.
But GlobalSign couldn't simply revoke the certificates with the private key in question.
"Because they were used for timestamping documents, if you actually revoke them, any certificates would be rendered invalid," Vermote said. "In our case, that would mean that a few million PDF documents signed with GlobalSign certificates would be rendered invalid, which we of course wanted to avoid."
As a result, Vermote and GlobalSign conducted a roadshow of all locations with those keys to review key management activities and destroy the keys in the presence of an official auditor. This included the three major locations where active key pairs are stored (locations with "passive" copies stored as backups do not require key ceremonies).
Earlier this month, Vermote had to visit one of the three locations where GlobalSign stores active key pairs because he is the designated key manager and ceremony master for the European location, which he requested not be disclosed. Vermote picked up a GlobalSign colleague early in the morning, drove across several countries to reach the facility, with multiple police stops and border patrol checkpoints that involved trunk searches, travel approval reviews and even some lecturing on social distancing.
"We were told at one point that my colleague was sitting too close to me and had to sit in the backseat of the car," he said. "We had to stop at another office on the way to pick up secrets stored in a safe that are required to unlock the key material."
When they reached their destination, the process was more complicated than normal because all personnel involved had to wear extensive PPE and use their own individual keyboards while maintaining safe distance from one another.
In total, it was an 18-hour day for Vermote. But he noted that key destruction is "very rare" and that GlobalSign is confident it can conduct key generation, revocation and even destruction activities "as normal" going forward -- as long as conditions don't worsen.
Audit delays and uncertainty
Audits are not as critical as key management ceremonies, but they're still an important part of certificate authority security. Failed or incomplete audits can have serious consequences for certificate authorities; lax audits were one of several issues for Symantec's CA business that led Google, Mozilla and other web browsers to deprecate thousands of the company's certificates in 2018 (Symantec sold the business to DigiCert in 2017).
The audit programs come in two types: the WebTrust program, which is run by the Chartered Professional Accountants of Canada and the American Institute of Certified Public Accountants, and the ETSI program, which is run by the European Telecommunications Standards Institute. But the ultimate decision-making authority regarding audits rests with the major browser companies -- namely, Microsoft, Google, Mozilla and Apple.
Some CAs such as DigiCert were fortunate to have already completed their WebTrust and ETSI audits earlier this year, before restrictions went into effect.
But other organizations were not as lucky. Vermote raised concerns about the auditing process in the mozilla.dev.security.policy forum on Feb. 19. After some discussions between CA representatives and officials from Mozilla and Google, the browser makers updated their guidance regarding audit delays.
"We've not seen any material impact resulting from the COVID-19 situation for now. We know it's likely that audits will be delayed because auditors won't be able to go on site and this is why we're giving guidance on the wiki," a Mozilla spokesperson said.
"When a CA realizes that their audits will be delayed by a force majeure, Mozilla expects the CA to promptly disclose the issue, to provide regular updates, and to remain fully compliant with all other aspects of the Mozilla Root Store policy," the company said on its wiki page.
So far, the browser makers haven't reported any issues stemming from the pandemic.
CAs and browser companies appear to be on the same page regarding audits and policy enforcement; Tim Callan, senior fellow at Sectigo, said he'd be surprised if Google, Mozilla and Microsoft don't offer some flexibility to CAs who make best-faith efforts to communicate and provide transparency.
"They have a legitimate need to know that CAs are doing their jobs correctly, and audits are a critical part of that. And at the same time, there's also a legitimate need for them to recognize that if you're physically not able to have a WebTrust auditor on site, for health and legal reasons, then that is that," Callan said.
But no one is sure how long the pandemic will last, and how far that flexibility may extend. France said Sectigo has discussed the idea of a "virtual" audit where third parties use secure video conferencing channels like ICANN used to monitor and review the CA's facilities.
While that approach may be technically feasible, the decision is ultimately up to the browsers' root programs.
"It's a question we don't have an answer to yet," France said, "and I don't even think the various auditors themselves have a good answer to it today because the situation is unprecedented."
I wonder whether those CAs can do emergency revocations, which might be required for key compromises and are critical to internet security, within 24 hours.
Arvid VermoteCISO, GlobalSign
The major CAs say the COVID-19 pandemic has demonstrated that their continuity plans are effective and that they are adequately prepared to weather the storm. "I think evidence of that work was done by all the major CAs is that we're not seeing any stories about major outages, and I think that's important," Callan said.
However, Callan said, if this extends to a year or beyond, then the CAs, web browsers and auditors will need to have discussions about changing some practices and expectations.
The situation could spark discussions about long-term policy changes for CA oversight. "I think there's a window of opportunity to put more into the standards to make sure these things can be prevented," Vermote said.
But perhaps the biggest immediate concern among experts is whether all CAs have strong continuity and disaster recovery plans. Some organizations may not have the infrastructure or personnel to quickly and effectively respond to emergencies.
While GlobalSign and Sectigo both said they are they are taking a more conservative approach to partnering with organizations during the pandemic, the digital certificate ecosystem is already vast.
"There are a bunch of smaller, regional CAs," Vermote said. "And I wonder whether those CAs can do emergency revocations, which might be required for key compromises and are critical to internet security, within 24 hours."
