Brown CS Blog

by Andrew Ayer

Click the link that follows for more Brown CS content about how our alums continue to innovate and pioneer the field.

Andrew Ayer graduated from Brown in 2012 with an ScM and ScB in Computer Science. He focused his CS studies on systems and architecture, taking courses such as CS 168 and CS 167/9, serving as a Head UTA for CS 31 (Introduction to Computer Systems), and helping develop CS 195S (Fundamentals of Computer Systems).

When you visit a secure website (the kind whose address starts with "https"), your web browser needs to be sure that it's really talking to that website and not to an impersonator who wants to trick you or steal your data. Much like in real life, when we ask to see a photo ID to verify someone's identity, your browser requires the website to present an SSL certificate that attests to its identity. Just like we trust government agencies to issue IDs only after validating the ID applicant's identity, your browser trusts a set of certificate authorities to issue SSL certificates to websites only after validating the identity of the certificate applicant, who is typically the website operator or domain owner.  

Unfortunately, sometimes this trust is misplaced and browsers have to revoke trust in certificate authorities, as happened this year to Symantec's certificate authority after I discovered 108 certificates which Symantec had issued with improper validation.

My foray into the world of SSL certificates started one night many years ago when I was the on-duty SPOC and received a problem ticket about one of the Department's email services being down due to an expired SSL certificate. I renewed the certificate and the service came back up, but this was not the only time an expired certificate caused downtime for Brown CS. With 60 certificates to manage and each one expiring every year, it was easy to forget to renew. I eventually wrote a script that scanned for certificates and emailed Problem one week before expiration, but it seemed like there ought to be a better way.

Two years after graduating, with my experience of being a SPOC on my mind, I founded a company called SSLMate to make SSL certificates easier through automation. I soon learned that automation wasn't the only thing the SSL certificate industry needed – there was also a troubling lack of accountability over certificate authorities. With the ability to issue certificates for any website, certificate authorities are rightly called the keyholders of the Internet. If they mess up and issue a certificate to the wrong person, that person can use the certificate to impersonate a website. When you visit your bank's website, don't you want to make sure it's really your bank on the other end?

Browsers impose rules on certificate authorities to ensure that, for example, only CIS can get a certificate for www.brown.edu, and browsers only trust certificate authorities that pass yearly audits. Firefox currently trusts 58 companies, government agencies, and nonprofits to be certificate authorities. Unfortunately, there are inherent limitations to audits, which focus more on documented procedures than making sure the certificate authority is behaving 100% correctly. Consequently, certificate authorities regularly violate the rules despite passing their audits.

Justice Brandeis wrote that "sunlight is the best disinfectant". In 2013, Google took this observation to heart and created Certificate Transparency, a system to publicly log all SSL certificates. Although Certificate Transparency can't directly stop certificate authorities from issuing bad certificates, it makes the bad certificates public and more likely to be discovered. Certificate Transparency is crowdsourced certificate authority auditing, and with more eyes upon them, certificate authorities have a greater incentive to behave. Google Chrome will only trust SSL certificates issued after April of 2018 if they're published in at least two Certificate Transparency logs.

I was drawn to Certificate Transparency not only for its potential to improve Internet security, but also because it's very exciting to work on such a new technology. Google invented Certificate Transparency, but it's being developed as an open standard by the Internet Engineering Task Force, and Chrome holds its policy discussions on an open mailing list. For the past two years, I've participated in both forums, and last year, I created Cert Spotter, a Certificate Transparency monitor that scans logs looking for the certificates for a domain. Domain owners can use a monitor to look for rogue certificates that they didn't authorize.

To demonstrate Cert Spotter, I set up a web page with a live demo showing the certificates for example.com, a real domain owned by ICANN (the administrators of the Internet's domain system) that's been set aside for documentation purposes. ICANN operates a secure website at www.example.com with a certificate issued by the DigiCert certificate authority. It was perfect for my demo: I set it up and forgot about it.

Six months later, I was looking at my demo and noticed there were four new certificates for example.com, issued by Symantec, and they looked rather odd. Three of the certificates were issued not to ICANN, but to a South Korean company named "Crosscert”. Another certificate was valid for products.example.com, which seemed unlikely for a domain reserved for documentation purposes. I started looking in Certificate Transparency logs for other, similar-looking certificates and found 104 more suspicious Symantec certificates, including a certificate issued to test.com (a testing company), and almost 100 that had been issued to a company named "test" supposedly located in "Test, South Korea".

On January 19, after confirming with ICANN's security team that they didn't authorize the example.com certificates, I published my findings to mozilla.dev.security.policy, a mailing list used by the industry to discuss certificate authority policy. Mozilla and Chrome immediately began an investigation and required Symantec to publicly answer a series of questions. Symantec tried to stonewall the investigation by delaying their answers and answering in insufficient detail, but as the browsers and other members of mozilla.dev.security.policy continued probing, the truth eventually came out. It didn't look good for Symantec.

Since 2013 or earlier, Symantec had delegated the authority to validate certificate requests to four organizations with barely any oversight, including a South Korean company named Crosscert. When Crosscert told Symantec to issue the certificates for example.com and test.com, Symantec issued them without validating that the owners of those domains had really authorized them. Symantec was trusting Crosscert to do the validation themselves, and although Symantec required audits of all four companies, the audits were incomplete, overdue, or downright awful.

As I explained, even perfect audits frequently miss problems. In this case, the audits stunk, yet Symantec took no action and failed to proactively disclose the audits to browsers. When Symantec finally began scrutinizing these four companies after my report, they found that Crosscert was keeping incomplete records, making it impossible to retroactively check that they had been validating certificates properly.

To make matters worse for Symantec, the investigation uncovered even more problems. On top of that, Symantec was already under scrutiny by browsers for incidents in prior years, including a discovery in 2015 that Symantec employees had been testing their production systems by issuing real, live certificates for domains without authorization. Mozilla assembled a report detailing all the prior and newly-discovered issues with Symantec, and it didn't look good.

On March 23, Chrome began a public discussion on how to distrust Symantec. This proved to be a hard problem. When a certificate authority is distrusted, all of its previously-issued certificates stop working, including those that were properly issued. Any website with a Symantec certificate would need to replace it with a certificate from a different certificate authority, or it would stop working.

Symantec was a huge certificate authority, owning 30% of the market in in 2015, and responsible for 42% of certificate validations in Firefox.  Never before had such a large certificate authority needed to be distrusted. Even worse, many of Symantec's customers were using certificates in such a way that they couldn't easily switch to a different certificate authority.

On July 27, Chrome posted their final plan to gradually distrust Symantec, beginning on December 1, 2017, and ending in September, 2018. Symantec would be allowed to become a certificate authority again in the future, but only by using brand new infrastructure with stricter oversight. In the interim, Symantec would need to become a reseller of another certificate authority to continue providing certificates to its customers. Mozilla followed with a substantially similar plan to distrust Symantec in Firefox.

Becoming a reseller of a competitor is a difficult pill for any company to swallow, and on August 2, Symantec announced they were being acquired by DigiCert, an established certificate authority with a far better track record. DigiCert moved Symantec's customers to DigiCert's existing infrastructure, and will be replacing existing Symantec certificates before they are fully distrusted next year. In the midst of all this, the CA/Browser Forum, the industry group which sets baseline rules for certificate authorities, instituted a new rule to forbid certificate authorities from delegating domain validation authority to third parties as Symantec had done with Crosscert.

I'm happy to say that the certificate authority ecosystem is more secure at the end of 2017 than it was at the beginning. A bad actor has been distrusted, and a bad practice forbidden.  Certificate Transparency continues to pay off – in September, it helped me find security problems in two additional certificate authorities, GRCA and EDICOM. GRCA fixed the problem, and EDICOM has been distrusted by Firefox at their request. Meanwhile, SSLMate continues to grow, and there has recently been a surge of interest in Cert Spotter as more companies learn about Certificate Transparency. And Certificate Transparency isn't even fully deployed yet – I can't wait until Chrome starts requiring it next April!

The views and opinions expressed above are those of an individual, and do not necessarily state or reflect those of Brown University or Brown University's Department of Computer Science, nor does their publication here constitute an endorsement of them.

For more information, click the link that follows to contact Brown CS Communication Outreach Specialist Jesse C. Polhemus.