The AI Challenge: Balancing Open and Closed Ecosystems
The US and EU grapple with AI systems control
Over the past year, ChinaTalk has been closely following the development of AI in China. This coverage, of course, is incomplete without input from mainland Chinese researchers and experts. Over the past three months, we’ve contacted at least twelve different AI researchers in China across leading universities and technology firms in the hopes of inviting them onto the podcast to discuss their work. Sadly, none of the outreach efforts has been fruitful.
This is perhaps unsurprising given the current geopolitical climate. But given that so much research remains open source and transpacific, and we are still optimistic about hosting a meaningful exchange.
So if you are (or know) an AI researcher, analyst, or entrepreneur in China who would like to share thoughts with our audience in any capacity — English or Chinese, on or off the record — please do reach out! I’m at jordan@chinatalk.media.
假如没有中国研究者和专家的参与，我们对中国AI发展的报道肯定会缺乏完整性。如果您是一位中国AI研究员或企业家，并感兴趣与ChinaTalk的听众们分享一些心得，请随时联系我们团队；我们会非常感激！
The AI Challenge: Balancing Open and Closed Systems
Pablo Chavez is an Adjunct Senior Fellow with CNAS. He has held public policy leadership positions at Google, LinkedIn, and Microsoft, and has served as a senior staffer in the US Senate.
Tech debates, AI included, often boil down to a tug-of-war between open and closed systems.
On one side, open allows interoperability, customization, and integration with third-party software or hardware. Champions highlight how openness promotes transparency, accountability, competition, and significant innovation. On the other side, defenders of closed systems argue that they are more stable and secure and better protect their owners’ property interests.
Last week, two leading United States senators tipped the US AI debate between open and closed toward the latter. Senators Richard Blumenthal (D-CT) and Josh Hawley (R-MO), the two leaders of the Senate Judiciary Subcommittee on Privacy, Technology, and the Law, wrote a letter to Meta CEO Mark Zuckerberg to express concern about Meta’s public release of LLaMA, a large language model (LLM) developed in-house by the company.
The senators were particularly troubled that the LLM was made “available to anyone, anywhere in the world, without monitoring or oversight” — in other words, their concern is that LLaMA is open source. While acknowledging the value of open source, the senators stated that closed, “centralized AI models can be more effectively updated and controlled to prevent and respond to abuse compared to open source AI models.”
Further pulling in the direction of closed, Senator Hawley has published legislative principles that, if enacted, would “require generative entities working on generative AI models to obtain a license.” This requirement would appear to apply regardless of the size or nature of the generative model in question. It’s possible that such entities could ultimately open source their models, but without more detail the licensing requirement seems incompatible with open source development and distribution of AI.
Though the two senators lean toward closed, much of AI’s creation and evolution have happened thanks to open-source development and diffusion. Numerous widely adopted AI open-source projects provide development frameworks and libraries such as PyTorch, TensorFlow, and MXNet; and many companies — including Hugging Face, Stability AI, Nomic AI, and Meta — have released open-source AI models or enable open-source development. And just in the last few days, the United Arab Emirates’ Technology Innovation Institute open sourced its Falcon 40B LLM — highlighting the fact that open source AI development is happening throughout the world.
Google and OpenAI have also traditionally stood on the side of openness. Both have published AI research and open-source tools. Google, for example, originally developed TensorFlow in-house and later released it as an open-source software library for building AI. But both have become more closed and protective of their models: “If you believe, as we do,” that “AGI [i.e., artificial general intelligence that essentially performs any intellectual task that a human can] is going to be extremely, unbelievably potent, then it just does not make sense to open source,” says Ilya Sutskever, OpenAI’s chief scientist and co-founder. Google’s Zoubin Ghahramani echoes this sentiment: “We want to think more carefully about giving away details or open sourcing code.”
As suggested by these moves from open to closed, AI policy answers will not always be fully open or fully closed. Instead, policymakers will, often appropriately, draw an AI regulatory line somewhere between the two boundaries of the open-closed spectrum — and perhaps that is the process that Senators Blumenthal and Hawley have started in Congress.
The debate over finding that balance is most advanced in Europe, where policymakers are moving forward with the AI Act. Recently, the European Parliament proposed amendments to the Act that suggest a hybrid approach to open versus closed. Concerned about the growing capacity of tools such as ChatGPT, the Parliament decided that “foundation models” may be open source but only in a rule-bound way. Foundation models are AI models — including large language models (LLMs) — that are “trained on broad data at scale, designed for generality of output [i.e., able to perform many different functions], and can be adapted to a wide range of distinctive tasks.”
The proposed parliamentary language would impose significant compliance requirements on open-source developers of foundation models, including the obligation to achieve “performance, predictability, interpretability, corrigibility, security, and cybersecurity throughout [their] lifecycle.”
Realistically, only an organized and well-funded — and perhaps European — open-source project could meet these obligations. Yet open-source projects could still build foundation models, at least in theory.
The proposed restriction attempts to thread the needle between the EU’s interest in supporting open-source AI — important, among other things, to the EU’s efforts to achieve strategic autonomy — and ensuring trustworthy AI development through legislative guardrails.
Whether or not this particular middle-ground outcome represents the right policy answer, it is an approach that uses the open-versus-closed framework to break the binary. By doing so, it prompts an important discussion about deep policy concerns.
Can semi-closed open-source ecosystems work? Will the European Parliament’s proposed guardrails for open-source foundation model development harm EU competitiveness? Will allowing open-source development of foundation models at all put powerful AI in dangerous hands and create a threat to the EU and beyond?
The last question is one that policymakers are grappling with by potentially drawing a line between open and closed. In testimony before a Senate subcommittee led by Senators Blumenthal and Hawley, OpenAI CEO Sam Altman stated that AI regulation should take effect above a capability threshold and suggested that open source should not be permitted above that threshold. Below that threshold, however, open source could continue. As Altman later tweeted: “We shouldn’t mess with models below the threshold. [O]pen source models and small startups are obviously important.”
In a possible move back toward open, OpenAI reportedly plans to open source one of its LLMs in the near future. The size and nature of that model — along with Meta’s LLaMA and others — will help develop the rationale for a future potential threshold (and, to be clear, much of this debate is and should be about future models rather than current ones). Below that level, open source would continue unfettered. Above it, AI models would be subject to restrictions, including licensing and testing requirements and potentially export controls.
The distinction between AI below the “capability threshold” and AI above that threshold will be difficult to draw. Already the conversation seems to be shifting from AGI to concerns about superintelligence (i.e., AI that is even more capable than AGI). The decision is also complicated by issues such as how and whether AI systems should align with human values and goals.
Yet the threshold question prompts policymakers and other stakeholders to ask which parts of the AI ecosystem should be open and which are more appropriate to restrict. It suggests a granular and more precise policy outcome: a largely open AI ecosystem that exists alongside a closely regulated system for foundation models above a certain capacity threshold. This approach would be similar to the performance threshold that divides advanced integrated circuits from standard ones in the Biden administration’s restrictions on chip sales to China.
The open-versus-closed framework remains imperfect. In the AI context, both sides have a legitimate claim to being a better model for safety and security. As noted by Stability AI, an open-source AI company, open models and datasets can help ensure robust oversight, and third parties with access can anticipate emerging risks and implement mitigations. But the nature of open-source licensing (which generally makes source code available to all) opens the door to entities or individuals who wish to cause harm or are simply not concerned about or resourced for risk mitigation. These are among the concerns expressed by Senators Blumenthal and Hawley in their letter to Meta.
Still, the open-versus-closed framework helps policy analysis because it results in nuanced and targeted solutions. Even for AI that is not fully open source, third parties such as academics and regulators should be allowed to review datasets, weights, and other critical components of AI models by third parties to help ensure the models’ safety.
Navigating the spectrum between open and closed will be important to AI policy development. Finding the right balance will help promote innovation while managing the very serious short- and long-term risks posed by the technology.
The European Parliament’s approach and OpenAI’s threshold model attempt to find an appropriate balance between open and closed. Neither is perfect. But both contribute to a much-needed debate to leverage the best of open and closed systems to ensure that AI evolves in a way that benefits and doesn’t harm humanity.
Editor’s Note: An earlier version of this article was published by Bandwidth, an online journal dedicated to advancing transatlantic cooperation on tech policy published by the Center for European Policy Analysis (CEPA). To read the latest from Bandwidth, click here.