OpenAI launched its GPT Store (Store) in early January 2024 to “help you find useful and popular custom versions of ChatGPT.” For companies, it promises new productivity with more AI-powered applications that employees can use with the organization’s own datasets, while enjoying the familiarity and built-in capabilities of ChatGPT.
For one set of employees – companies’ overworked technology compliance teams and cybersecurity engineers – the arrival of a third-party app store full of generative pre-trained transformers (GPTs or apps) poses an unnerving challenge. In three months, OpenAI claims, the Store has accumulated three million custom GPTs based off its large language model (LLM).
The new GPTs can link a series of actions for users to “allow GPTs to integrate external data or interact with the real-world [sic],” Open AI highlighted. That means that the user’s data often will travel to a developer’s third-party server. In a sample of the most downloaded productivity GPTs, roughly a quarter provide the option to upload files, Harmonic Security CEO Alistair Paterson told the Cybersecurity Law Report.
The Store provides “a thin veneer of legitimacy around some of these applications, but fundamentally, the store’s not really checking where the data is going and how it’s being secured,” Paterson said. Now that many companies use ChatGPT’s Enterprise version, their employees could get a false sense of security about the legitimacy of the new apps that work with it. “It is in no way like Apple’s App Store ecosystem, where there is a very heavyweight check that apps have to go through before they get listed,” he cautioned.
This article details the top security risks of the Store and identifies key priorities for compliance professionals and company engineers as they confront the risks of the new GPT app ecosystem. It also suggests resources for cyber compliance professionals to monitor security concerns around LLMs more broadly.
See “Dos and Don’ts for Employee Use of Generative AI” (Dec. 6, 2023).
The Short Path to a Store With Millions of GPTs
In 2023, a few months after ChatGPT emerged publicly, OpenAI allowed outside developers to offer plugins as add-on tools to it. These extensions allowed users to go beyond the main trained LLM to fetch real-time data or information on the web, then perform a chain of actions. By the end of 2023, almost 1,000 plugins were available for the platform.
Developers told OpenAI it was easier to build standalone GPTs trained on its LLM than to create plugins. At the end of 2023, OpenAI shifted direction. It encouraged outsiders to build self-contained GPTs preloaded with its DALL-E, Code Interpreter and ChatGPT features, setting the Store in motion. OpenAI will shut down all “conversations” using plugins on April 9, 2024.
The arrival of standalone GPTs will let developers in different industries leverage the LLM for structured tasks specific to its business operations. The Store offers “custom versions of ChatGPT that combine instructions, extra knowledge, and any combination of skills.” These will allow actions “such as connecting GPTs to databases, plugging them into your emails, or making them your shopping assistant,” the Store highlights.
See “Apple Overhauls Privacy for iPhone Apps, but Will It Enforce Its Policies?” (Sep. 23, 2020).
The GPT Store’s Structure and Its Appeal to Business
The Store’s approach is similar to Apple and Google’s lucrative mobile app stores. It displays leaderboards and curated picks. It offers developers a payout based on how many people use their GPT. “We believe the most incredible GPTs will come from builders in the community,” OpenAI says.
Developers, including those at companies, can set a GPT’s visibility to “only me,” “anyone with a link,” or “everyone.” Some GPTs will work only with OpenAI’s three paid tiers, called Teams; Plus; and Enterprise.
One difference from the marketplaces for mobile apps or smart speaker apps is that, with the GPT market, businesses are expecting to gain powerful new productivity tools with AI.
The Store’s Low Barrier to Entry
OpenAI’s GPT Builder lets most anyone program a GPT. A GPT maker uses regular language, rather than computer code, to tell the tool the desired capabilities and actions for the new GPT, and then it spits out the GPT to enact those functions.
OpenAI emphasizes the Store’s benefits for ChatGPT Enterprise customers, saying its GPT builder tool will “empower users inside your company to design internal-only GPTs without code and securely publish them to your workspace. The admin console lets you choose how GPTs are shared and whether external GPTs may be used inside your business.”
All creators must verify their profiles, but it is possible that, along with trained engineers, Instagram influencers, YouTube stars and scam artists will make a GPT.
“We use a combination of automated systems, human review and user reports to find and assess GPTs that potentially violate our policies,” OpenAI says. A violation “can lead to actions against the content or your account, such as warnings, sharing restrictions or ineligibility for inclusion in GPT Store or monetization.”
Since January 2024, the Store’s process for vetting creators remains unclear, Paterson reported.[1] “I am incredibly bullish on the overall AI revolution for businesses. There are a ton of exciting applications, but I am not sure that the GPT Store is going to be a significant part of that,” he opined. So far, “it looks like mostly a series of thin wrappers on top of a groundbreaking technology, which is ChatGPT. But these wrappers are not adding a ton of value on top. The value likely is going to come from the more thought-through, independent applications that are now coming out for enterprise every day,” he observed.
Practically, though, employees will find many of the new external GPTs compelling. “You can automate the boring aspects of your job and potentially make yourself look better and faster to your boss,” Paterson pointed out.
Thus, businesses soon will have to set fresh guidelines about the Store and its many GPTs, decide whether to permit use of any Store GPTs, and educate employees. To do that, businesses will need to understand the extent of the Store’s risks.
See “Go Phish: Employee Training Key to Fighting Social Engineering Attacks” (Aug. 9, 2023)
OpenAI’s Developer Rules
OpenAI has taken some steps to require GPT developers to protect privacy, security and safety, but observers see notable gaps and flaws in its rules three months in.
OpenAI Flags Privacy
OpenAI lists privacy first in its policies for builders and prevents them from accessing users’ chat with their GPT. But some user data still will travel. If a GPT is programmed to use third parties’ application programming interfaces (API, a site’s code gateway for secure communication), the developer selects whether the user’s chat can be sent to each API.
OpenAI’s rules direct GPT makers to not compromise the privacy of others, including “collecting, processing, disclosing, inferring or generating personal data without complying with applicable legal requirements.” Nor are GPTs supposed to solicit “sensitive identifiers, security information, or their equivalents: payment card information (e.g., credit card numbers or bank account information), government identifiers (e.g., SSNs), API keys, or passwords.”
OpenAI also forbids makers from using facial recognition or other biometric systems for assessment or identification. Unauthorized monitoring of individuals is likewise banned.
See “Checklist for Framing and Assessing Third-Party Risk” (Aug. 16, 2023).
Less Guidance and Controls on GPTs’ Security
OpenAI’s rules’ attention to cybersecurity is milder. For each action a GPT takes involving a party, the developer must supply a privacy policy URL and decide whether to require authentication with API Key or OAuth, but the developer also can select “none.”
One piece of good news is that OpenAI’s centralized process and easy-to-use app creation should reduce some of the “shadow AI” that festered throughout 2023, Paterson said. Outside of OpenAI, over 10,000 third-party apps popped up to offer appealing generative AI uses, sometimes building on ChatGPT’s capabilities. Many had poor functionality and dubious security and privacy measures, he noted.
The Store does not yet offer much documentation on cybersecurity, observers have lamented. OpenAI seems to have a team working on it, Paterson said, so that may change.
See “Navigating NIST’s AI Risk Management Framework” (Nov. 15, 2023).
Prohibitions on Cheating, “Jailbreaking” and IP Circumvention
OpenAI’s rules for GPT makers highlight its many other headline controversies more than security risks. Its prohibitions reflect the copyright lawsuits that OpenAI is fighting, deepfakes, political disinformation and widespread attention to ChatGPT’s inappropriate output, like abusive content, sexual material, and discriminatory automated decisions.
Plenty of GPTs in the Store raise copyright concerns, Paterson noted. Some let users speak in the voice of trademarked characters or create art in the style of copyrighted material.
Other GPTs on offer purport to let users prompt ChatGPT to generate nasty or violative output that the LLM is programmed to avoid. Some Store GPTs also circumvent integrity tools such as plagiarism detectors and AI content detectors like Originality.ai.
See our two-part series on managing legal issues arising from use of ChatGPT and Generative AI: “E.U. and U.S. Privacy Law Considerations” (Mar. 15, 2023), and “Industry Considerations and Practical Compliance Measures” (Mar. 22, 2023).
Security Risks for the GPT Store
Lack of Security By Design
“The way that OpenAI designed its platform had very obvious issues,” Washington University security researcher Umar Iqbal told the Cybersecurity Law Report.
The ecosystem’s architecture is insecure. “Running a lot of apps in the same execution space is not a thing anymore in modern computer security – when a platform opens an app, it sandboxes it using some virtual restrictions. OpenAI did not do that,” Iqbal noted.
OpenAI has left it possible, when users run multiple GPTs, that the GPTs can steal and overwrite each others’ stored files, Iqbal noted. He cited another researcher’s report of this vulnerability (in February 2024) with OpenAI’s Code Interpreter, which employees will want to use to analyze data and files written in code. The thefts could include the user’s conversations with ChatGPT, the report warned.
The “multi-tenant” environment and sharing of computing power and infrastructure that OpenAI has established is a general risk, Paterson added, “but that is probably the only way the GPT Store would ever work economically.”
See “Innovation and Accountability: Asking Better Questions in Implementing Generative AI” (Aug. 2, 2023).
Third-Party Interactions and Chances for Stolen Credentials
The arrival of GPTs that perform a chain of actions will require increasing interactions across websites and dependency on third parties, often running multiple apps simultaneously, Iqbal said.
This poses “a huge third party risk management challenge that companies have now on their hands,” Paterson observed.
Whitelisting GPTs would require labor-intensive vetting. For example, one popular GPT in the Store currently analyzes PDFs, requiring users to upload their files, which goes to a third-party site, Paterson noted. That site does not say it is GDPR, ISO 27001 or Soc2 compliant, nor does it indicate that it has “any real security program around the data,” Paterson reported.
A GPT performing actions across sites and apps will need authentications to enter multiple user accounts. That makes it plenty appealing for malicious GPT operators, noted a report from Salt Security. The report provided examples of the relative ease of stealing security tokens and passwords in the process.
See “Checklist for Building an Identity-Centric Cybersecurity Framework” (Nov. 3, 2021).
Unintentional Conflicts and Data Leaks
GPT apps “can cause inadvertent security, privacy, and safety issues” because their instructions sometimes redirect the LLM, Iqbal noted. On March 7, 2024, during the FTC’s PrivacyCon, Iqbal presented research evidence that vulnerabilities with ChatGPT plugins persisted throughout 2023.
One inadvertent problem Iqbal found involved personal data. For example, both a medical appointment GPT and a travel reservation GPT will gather personal information to complete its task, but “the LLM can get confused about what is the personal data needed by each of the apps,” and inadvertently upload sensitive health data into the travel app, he reported.
With the lack of isolation in the ChatGPT environment, plugins in 2023 also hijacked other apps’ sessions. An initial instruction “sometimes persists beyond the context of using that app,” Iqbal said. In one example, an app switched the language for the LLM’s response, and then the LLM continued to use that language in other apps, deviating from the user’s usual language.
The Risks of Natural Language
Generative AI’s appeal lies in the flexibility of natural language inputs, but those can be vague compared to the engineered code in web and mobile ecosystems. “The natural language interface of a GPT is an additional attack vector, because of the ambiguity and imprecision,” Iqbal cautioned.
Natural language made it easier to confuse the LLM, noted University of Washington assistant professor Franziska Roesner, who collaborated on the research with Iqbal. One app could affect other apps with certain instructions. In one travel app, “the natural language description said something in capital letters like ‘always use this for travel-related queries.’” With two similar apps installed, “ChatGPT picked the one that yelled at it, basically,” she said.
OpenAI has responded to alerts of vulnerabilities, but Iqbal has spoken to developers who reported vulnerabilities with their plugins that were not remedied. “It has been only three months since GPTs have launched and there are millions of GPTs. Similar to how OpenAI is moving fast on the innovation, they also need to move fast for securing their systems. Which is not as fast as that innovation at the moment,” Iqbal contended.
See “A 2023 Cyber Regulation Look-Back and 2024 Risk-Management Strategies” (Dec. 13, 2023).
Vetting and Procurement Steps
Revisit Gen AI Policies
Companies spent 2023 grappling with use policies for ChatGPT and other LLMs. The arrival of many narrow third-party GPTs is a shift from the previous focus on a few dominant LLMs, Roesner noted. “Now is definitely the time to think about how to vet these apps, how to design the APIs, how to think about permissions for user data,” she said.
“The only way enterprises can realistically manage [the array of threats] is to get the enterprise edition of ChatGPT,” and use its security settings, Paterson opined.
To address any employee interest in the GPT Store, or GPT apps independently available, run a fresh exercise with departments to identify their latest use cases for AI, Paterson suggested. “Then you can come up with some standardized, structured ways of meeting the business’s needs” and prevent individuals from using an assortment of unvetted GPTs, he said.
Companies might remind employees to be cautious about using sensitive material (business or personal) with any newer, “smaller” GPTs derived from ChatGPT, Paterson recommended. Meanwhile, organizations should take technical steps to verify “that sensitive data isn’t getting out of the business” and maintain ways to see “which applications are being adopted.”
See our two-part series on the practicalities of AI governance: “AI Governance Gets Real: Tips From a Chat Platform on Building a Program,” (Feb. 1, 2023), and “AI Governance Gets Real: Core Compliance Strategies” (Feb. 8, 2023).
Model the Threats to Address Other AI Risks
The GPT Store is not the only vector multiplying LLM risks for enterprises. New AI productivity software on the market promises seamless AI integrations. One product claims that it “allows you to access AI at any time, within any software.” It also features add-on plugins – possibly another layer of interaction between GPTs.
Companies’ cyber teams should undertake a fresh round of threat modeling that incorporates such apps and their possible interactions, recommended Iqbal.
A period of accelerating adoption may be coming and will shift the risks. Companies are still slowly testing Gen AI services and integrations, but “over the next 12, 18 months, they have to be adopting these tools or risk being left behind,” Paterson predicted. Many companies are now vetting Microsoft’s Copilot, he added, which “is a very different thing from the GPT Store, but also has enterprise implications because it is possible for employees to build their own Copilots with sensitive data and then make those Copilots available externally.”
To cope with the evolving risks, compliance teams will want to keep perspective and monitor broader LLM problems. One guide is the OWASP Top 10 for LLM Security. Iqbal also recommended the LLM Security site that collects reports – and some remedies. OWASP’s first-listed security threat is prompt injections, or malicious prompts. “This falls under those categories of attack for which we need research to come with effective solutions,” Iqbal said.
[1] OpenAI did not respond to multiple messages that the Cybersecurity Law Report sent to its communication and legal representatives to discuss the Store’s vetting and security measures.