Brown CS Blog

Guest post by Bill Marino, MSc Cybersecurity student.

 

In 2022, generative AI went mainstream. Mere months ago, it still seemed exclusively the province of the ML research community and of Twitter, where meme accounts like @weirddalle shared the results of feeding text-to-image generation models off-the-wall prompts like "a bottle of ranch dressing testifying in court."

Cut to the present, where generative AI startups like Stability AI and Jasper are raising $100 million rounds of funding, big players like Microsoft have announced upcoming integrations of text-to-image generation, and perhaps all of us have used OpenAI's DALL-E 2 to create something delightful like this "chicken nugget dressed as a vampire for Halloween".

Just the other week, Adobe, where I'm a Product Manager focused on generative AI, joined the party by announcing its first text-to-image generation feature. In parallel, President of Digital Media David Wadhwani ('93) announced that, as Adobe invests more in generative AI, it'll be doing so in an artistic-centric manner. 

  

But what many don't know is that the generative models behind all this dazzling art can also have a dark side. In particular, they have given rise to a new breed of cyberattacks. These attacks exploit the defining property of generative models —their ability to generate plausible new examples of some type of data— to synthesize passwords or lookalike fingerprints in order to break authentication, disguise malware as benign software in order to evade detection, and more. 

As a member of Adobe's Sustainability Committee, covering responsible AI, I have kept a close eye on these evolving malicious use cases. Eventually, I decided to enroll in the Cybersecurity ScM program at Brown so that I could study these attacks more closely. 

At Brown, I met computer vision professor James Tompkin, who had done work on detecting a related misuse of generative AI: deepfakes. An independent study with James led us to conclude that there was a need for a comprehensive systematization or taxonomy of generative machine learning-powered cyberattacks. So far, none exists. 

The taxonomy we have begun developing to fill this void is based on the observation that cyberattacks powered by generative machine learning —as opposed to other types of machine learning, such as discriminative— display a reoccurring set of attack patterns. Taxonomizing generative cyberattacks according to these attack patterns nets a streamlined and scalable systematization that not only helps researchers identify patterns across seemingly disparate generative cyberattacks but also helps them ideate undemonstrated threats and even potential defenses inside each attack pattern. 

For example, Hitaj et al. used leaked passwords to train a generative model that synthesized high-quality password guesses in order to break password-based authentication. Meanwhile, Bontrager et al. used fingerprint images to train a model that generated plausible new fingerprints, then located the ones that triggered false acceptances for multiple identities during fingerprint-based authentication (so-called "masterprints"). On the surface, these cyberattacks may seem quite different. But we believe they hew closely to the same attack pattern: using a generative model to recover real world examples (or near neighbors) in order to break authentication. Identifying the attack pattern that these two cyberattacks share lets us, in turn, observe other notable qualities they share. For example, these two cyberattacks have a comparable adversarial model, with attackers about whom we make similar assumptions (access to data and a model) and who have similar goals (breaking authentication) and capabilities (the ability to query the authentication system). Likewise, the defenses against these two cyberattacks are also comparable: from restricting access to training data to throttling access to the authentication system and more. If we know that generative cyberattacks with shared attack patterns also share adversarial models, defenses, and other features, researchers can therefore ascertain many important qualities of a new generative cyberattack found in the wild merely by classifying its attack pattern. 

On October 31, I presented an overview of the in-draft taxonomy at the  Brown Visual Computing Group weekly meeting (Brown users can view the Panopticon recording here). With the Topics in Computer System Security seminar course (CSCI 2951-E taught by Lilika Markatou and Roberto Tamassia) as my springboard, I aim to refine the taxonomy this Fall before submitting a systematization of knowledge (SoK) paper in early 2023.

The first image above (“brown university on a beautiful autumn day neon cyberpunk synthwave oil painting”) is by Bill Marino/DALL-E, the second (“a bottle of ranch dressing testifying in court”) is by Weird Dall-E Generations/DALL-E, and the third (“chicken nugget dressed as a vampire for Halloween”) is by DALL-E 2.

The views and opinions expressed above are those of the author, and do not necessarily state or reflect those of Brown CS, nor does their publication here constitute an endorsement of them. Brown CS has no financial involvement in any of the companies mentioned above and has not been compensated in any way for this story.

For more information, click the link that follows to contact Brown CS Communications Manager Jesse C. Polhemus.