Sovereign AI: Vanity Project or Natsec Necessity?
Sovereign AI: Vanity Project or Natsec Necessity?
Plus: How China started all of this, Meta's open source strategy, and detours into Canadian, French, and Singaporean AI
Good AI is good and bad AI is bad, but how do lawmakers tell the difference? Will AI bring the world together or balkanize the internet beyond repair? Why do governments even need cloud computing anyway?
To discuss, ChinaTalk interviewed Pablo Chavez, an adjunct senior fellow at CNAS and former Vice President of Google Cloud's Public Policy division, as well as the inestimable investing tycoon Kevin Xu. Xu, formerly of GitHub, is the founder of Interconnected, a bilingual newsletter on the intersections of tech, business, investing, geopolitics, and US-Asia relations.
In this interview, we discuss:
The digital sovereignty movement and the lessons we can learn from China's Great Firewall;
The value and risks of open source architecture in the future of AI governance;
Meta’s long history of open source and how Llama fits into that strategy;
The geopolitical and cultural forces driving nations to pursue their own AI strategies;
The viability of sovereign AI initiatives in the face of global tech giants.
Also, I’ll be in DC next week—reach out if you want to connect.
The Origin of AI Sovereignty
Jordan Schneider: Pablo, kick us off. Where does the story of Sovereign AI begin historically?
Pablo Chavez: It begins with the digital sovereignty movement in China in the late 90s and early 2000s. Essentially, this was an effort to push back against the internet freedom movement promoted by the US. That promotion started during the Clinton administration and has continued to this day under the Biden administration.
The Chinese government said, “Your right to promote content in our cyberspace is restricted. We have full sovereignty over our internet.” This went from a political slogan to a much more concrete policy on many fronts.
There was, of course, content blocking and filtering, but one of the most interesting things that the Chinese government did around the 2010s was restrictions on cloud computing, and specifically American cloud computing companies.
As Microsoft and AWS started to build cloud infrastructure throughout the world, the Chinese government required American companies to enter into joint ventures with a local partner. They had to give up part ownership of the physical infrastructure, give up some ownership of the IP, and essentially operate under a much more constrained system of control than what existed for cloud providers in other countries.
Importantly, it was also around the time that much of the digital world was starting to move into cloud infrastructure. Cloud data centers started to be a much more critical point of entry for the digital world than it had been before.
That was the first manifestation of cyber sovereignty, known now as digital sovereignty. However, the movement wasn’t limited to China. Perhaps because of the one-two punch of US-China tech competition as well as COVID, digital sovereignty in Europe really started to take hold around 2020. We started to see very concrete manifestations of European governments starting to take a little bit of the China playbook and apply it in Europe.
One specific example is France. Around 2020, the French government started to require sovereignty controls for foreign cloud providers, specifically for tech giants like Microsoft, Google, and AWS. If a foreign cloud provider was going to provide services to strategic enterprises or the French government, these controls required a certain percentage of the project to be owned by a French entity.
This concept, which originated in China, morphed into a more global and concrete policy for the control of digital technology. That ultimately became the foundation for what we now call “AI sovereignty.”
AI sovereignty is a policy of attempting to control the AI stack on a nation-state level— including models, technical infrastructure, and data. The Cloud is part of that technical infrastructure, so it's a core component of the AI sovereignty conversation.
Jordan Schneider: It’s interesting that China was ahead of the game in seeing the strategic importance of information, the internet, and the cloud.
But what has happened in China — which hasn't really happened in the rest of the world — is that China basically has its own domestic cloud ecosystem with Alibaba Cloud and Baidu Cloud. Countries like France and Saudi Arabia can't compete on a one-to-one basis with AWS, Azure, and Google Cloud, and that doesn’t appear to be their goal.
It’s tricky because you still need cloud infrastructure to have a modern economy. How do these governments plan to balance sovereignty vs the ease of doing business?
Pablo Chavez: Countries do want to exercise control over generative AI. They also recognize that they can't build the entire stack. What they're building is what you could call a “Jenga AI tech stack.” They own part of the stack — perhaps they might own the model, perhaps they might own the data.
But in terms of the infrastructure itself — including the cloud and the advanced chips necessary to develop and deploy AI — they're realizing that it is really much more of a partnership between the government and the (foreign) private sector.
As I mentioned with France, despite the fact that there is a public-private partnership with respect to the technical infrastructure, there's still a desire to put in place controls so that ultimately, it is the government or the domestic customer that has control over the workloads that are operating in the cloud.
The Case for AI Sovereignty
Jordan Schneider: I can see a lot of arguments against this. It seems like it could be wasteful and expensive and harm domestic innovation. These countries are realistically never going to get enough scale to compete.
Kevin and Pablo, I’d love to hear your case for AI sovereignty in contexts outside of China.
Kevin Xu: On the topic of sovereign AI, I actually went through a personal 180 recently.
When I first heard the phrase, it was in the context of Jensen Huang of Nvidia trying to expand the total addressable market of Nvidia. The motivation seems very obvious. Every nation needs to build their own AI, they're all going to use Nvidia GPUs, etc.
But then over time, I think about some of the common problems that the current state-of-the-art AI exhibits, one of them being hallucination, which I think we all know about. Essentially it’s the AI bs-ing their way through something it doesn't actually know or understand or hasn't codified per se.
That’s a layman's point of view on hallucinations. If I can be a little controversial and stir the pot here, it sounds like stereotypically American confidence when it comes to how ChatGPT wiggles its way through a question that it doesn't know anything about. It sounds very confident in not knowing.
Of course, I'm not saying all Americans are like that. We have many humble Americans all around the world who exhibit a lot of good values on that front. But the reason I think that is because I believe the current state-of-the-art AI model is codifying dominant stereotypes.
The dominant stereotype of global internet culture is, for the most part, a lot of American-generated content on Reddit, message boards, Twitter, etc. If you think about it from another country's perspective, say, India or Japan, they all have very strong national cultural identity. They will find that objectionable.
For them, there is a very strong cultural incentive to at least codify the kind of attitude, the kind of intelligence, that they find to be most representative of their point of view. We can talk about what that means when it comes to censorship, data control, or data governance in a bit.
For me, I did a 180 once that light bulb switched on in my head. There is a very strong non-technical motivation for many countries to want to have some version of sovereign AI.
I'll leave one personal anecdote.
Pablo, you've worked at GCP before. I used to work at GitHub, which is a Microsoft-owned entity. My day job was to promote GitHub's international expansion strategy. That was my job, to articulate why GitHub should expand in countries like Brazil, Japan, etc.
Initially, when I came into that gig, I thought, “These are open source developers, they all speak pretty good English, they all read code, we all think the same — so this job will be easy-peasy.”
That was not at all the case.
I spent a lot of time doing the groundwork for expanding in Latin America. Countries like Brazil and Argentina all require or desire some level of localization. Some of the localization has to do with laws on doing business, such as currency exchange, rules on how to access revenue, the tax code, that sort of stuff.
But a lot of that is actually cultural as well. They want to have some version of Portuguese in the interface if they can. They want to have a lot of educational content in their language, in their own developer and hacker culture, if they can. Even though the dominant hacker culture is very much American, and to a certain extent European-led, they desire this. We’re talking about the most connected, most technically sophisticated crowd in the entire world: people who code on GitHub every single day in open source.
If that crowd wants their own version of something, I can certainly imagine other parts of the population, or certainly the government, wanting their own version of technical intelligence. This was all before generative AI.
So now that we're in this sovereign AI world, I do think there is a good reason for sovereign AI to exist. Whether you think it's wasteful or not is a different question altogether.
Pablo Chavez: There's a gentleman named Nathan Benaich, from Air Street Capital, who was recently quoted in The Economist saying that government investments in AI are wasteful.
Perhaps, but you could say the same thing about venture capital. 90-95% of venture capital investments fail. That doesn't make the endeavor misguided and it doesn't mean that good can't come out of it.
I want to echo a point that Kevin is making. If you look through the documentation of the various sovereign foundation models and LLMs that have been popping up around the world, the language and cultural points are very strongly made.
One that comes to mind is the SEA-LION LLM that's being developed in Southeast Asia, led by the Singaporean government. It stands for "Southeast Asian Languages in One Network." The person who's responsible for the project basically said, "Look, we have 11 major languages in the region. Those are not necessarily reflected in American commercially available generative AI products. What's more, our regional culture isn't necessarily reflected."
It’s funny because in the quote, this person says, "We're not woke, and those products are woke." It's just this interesting combination of protecting language and culture that I think has a lot of validity, as Kevin explained.
Honestly, from my perspective, I'm still wondering whether sovereign AI is worth it and whether it is feasible in the long term. But it is definitely being attempted right now.
My own interest in sovereign AI comes out of this quote from Kai-Fu Lee from 2018 that I've thought about a lot during my time at Google and since then. Kai-Fu is a very smart individual, venture capitalist, technologist, a key player in both Google China and Microsoft China.
What he said at the time was essentially that AI would be a two-player, dyadic game between the US and China. At the end of the day, the US and China would have dominance. They would be generating the most revenue from AI and every other country in the world would essentially have to line up with China or with the US essentially as vassal states to those two countries.
That idea has really motivated a number of governments to say, "No, this is not how it's going to play out." They're making efforts to make it more of a multiplayer game than it otherwise would be. Whether you believe this is a long-term trend or not — or whether it's a good trend or not if you're making a normative judgment on it — looking at sovereign AI does really complete the picture of the geopolitics of AI.
A lot of conversations these days are about the governance of AI, responsible AI, and the constraints on AI. The general notion of sovereign AI is much more about the build side. The point that Emmanuel Macron made last year was, if you're not building AI and you're only regulating it, then you're losing out as a country.
Jordan Schneider: A few points to pick up on this. I'm sold on Kevin’s idea that AI is important for countries. A lot of governments around the world feel burned that Google and Facebook basically got to set the rules of the road for how the social media age is regulated. They're not comfortable with that happening again.
At what level of the stack are you focused on being sovereign? At what level of the stack is it realistic to be sovereign? From the data center perspective, if Google just has a VIE and is running some data centers in your country as opposed to the country next door, that's maybe nice. You get a few jobs, but I don't think you can exert a ton of leverage there.
Singapore buying 5,000 GPUs and training a little thing to make sure that some local languages get a little more representation seems nice. But it’s also the sort of thing where you could just set some access requirements and make GPT-5 score at 95% in your local language like it does in English. Then they'll train on the data far better and more efficiently than your government employees ever can.
Maybe there's something to Macron's idea of, "if you're not building, you can't really understand it, you don't have any skin in the game." But coming back to Pablo’s about this being a dyad and gains to scale. That's definitely the broader trend we're seeing with Llama, Anthropic, and OpenAI.
On one hand, there are only going to be a few players with enough GPU access and money to keep scaling. But on the other hand, if frontier AI just turns out to be this commodity product, then maybe there actually is this whole new universe in which more localized companies can play and apply the gains to their markets. In which case, maybe it isn't a big deal that OpenAI isn't Singaporean or French or what have you.
Kevin Xu: There are enough tools for current governments to satisfy their own definition of what sovereign means to them. None of it is necessarily economically driven. The Air Street guy Pablo mentioned said that it's a total waste of spending and frankly, anyone in the investment world who is preaching free capital economics thinks it’s a waste of money.
My working analogy for why governments still want to do this is hosting the Olympics or the World Cup. There's always this massive overbuilding of infrastructure: stadiums, roads, cool gadgets they want to show off to the rest of the world during those moments.
That is partly economically driven, at least that's always the justification, but so much of it is actually long-term cultural or pride-driven. It has very little to do with the economic return of those investments. Yet every single country wants to do them.
Pablo Chavez: I’ll take a slightly different perspective on the Olympics point of view, although I think there's a lot to that. The Australian science agency just came out with a report that articulated a strategy for the Australian government to deploy sovereign AI.
One of the main justifications the agency gave in its report for developing sovereign AI was the concern, “What if we get cut off?” Whether this is a commodity technology or not, ultimately there's a broad belief that AI is going to be a fundamental technology.
What happens if a foreign provider cuts off a domestic market? What if the prices change dramatically such that it's economically less feasible to use foreign AI products? What if the product itself changes, such that it's not reflecting democratic ideals, principles, and values?
What the Australian government is saying is, "Look, we need to have at least something in reserve that we can turn to in case we are cut off." Candidly, this is a broader concern throughout the digital world. It's something that's impacted cloud sovereignty, data residency requirements, and so forth.
Even if you're not looking at it from the perspective of economic ROI, in some ways it just ends up being more of a demonstration of capacity. It is a fallback position. It’s a way to protect a country if at some point in the future technical resources, products, and services get cut off from that country.
Kevin Xu: Pablo, what you're building on was the second part of my answer to Jordan's previous question. There’s this hedging aspect to sovereign AI that every country wants to build.
There isn’t a free market competition between the government-approved sovereign AI version versus the best-in-class, frontier models that OpenAI, Microsoft, Google, and others are going to roll out. The reason why they want to have that is because they want to have a minimum viable version of AI that they can control, because they probably lost out in Internet 1.0 and 2.0 in having any say in the development of the technical economy around the world.
They also just want to have enough capacity in certain industries where the government has a high incentive to have some say over. There are always regulated industries in every single country. When it comes to healthcare, strategic resources, legal, etc., having some government approved facility that has a sort of government-registered model that companies can build off of is actually very helpful. It codifies some of that so-called dominant or mainstream perspective of culture that I talked about previously.
So to your point, Pablo, the sovereigns aren’t trying to compete with the best-in-class companies. They just want to have a little bit of market share in this new economy that we're all still trying to grapple with in terms of how big it can be and what it can actually become for the rest of human civilization.
Open Source Strategery and Lessons From Dwarkesh
Pablo Chavez: On the model development side, the trend really has been leveraging existing open source models, fine-tuning them with not that much money, and deploying them often open source as well.
It’s just fascinating. Open source models have really fueled a lot of the AI sovereignty movement. One of my favorite examples of this is the TAIDE model in Taiwan, which was built for roughly $7 million.
TAIDE stands for "Trustworthy AI Dialogue Engine." The reason that the Taiwanese government supported the development of this model is because they're very concerned about Chinese government interference in Taiwan and over-reliance on Ernie Bot.
In fact, there have been press reports about how, in the wake of the Taiwanese presidential elections, you could go to Ernie Bot and ask, "Who's the new president of Taiwan?" It would give you the right answer: Lai Ching-te. But then it would add, "But it doesn't matter because there's only one China."
So the Taiwanese government acted, again in the spirit of having an alternative and a fallback. I want to emphasize the point that it was based on open source technology, not difficult to build, and not super expensive. This is the fallback technology that's there in order to guarantee that there isn't manipulation of the political system, society, and so forth.
Kevin Xu: That's right. To add one more thing to what you said, Pablo, it makes me think of the grounding technique that is very necessary when it comes to all AI model training.
One thing I remember from a few weeks ago was when your old employer Google Cloud Platform had their annual conference. They're now providing grounding based on Google search for models trained on GCP. What grounding there really means to me, is basically saying, "We know the truth, we know the facts. If your AI goes off the rails, we provide the service to help you ground that AI."
When you expand that into sovereign AI, the example you used with Taiwan is very instructive. Different countries do have their own version of AI "truth.” Every country needs its own grounding, in that sense.
You don't need a lot of money to have sovereign grounding AI for your culture. It's not supposed to be competitive with the marketplace. It just has to be this almost archival knowledge encoded in the best-in-class model with the right data. You have a place to ground your culture or your country when misinformation flies in the air, which is going to happen on some level in the future with generative AI.
Pablo Chavez: One other thing, Jordan. You were focused on the model part. Not to put words in your mouth, but I think that maybe you were suggesting something like "Oh hey, what's the big deal here if at the end of the day the technical infrastructure —- the data centers, the chips inside of those data centers, etc. — don't belong to the country or government and instead belong to a foreign provider?"
As you look at the AI stack, there are definitely examples of governments trying to not just partner with, for example, American cloud companies on deploying that part of the AI stack, but also actually trying to take sovereign resources and deploy those to train models.
For example, France has the Jean Zay supercomputer. A number of other countries have supercomputer capacity that they're now retrofitting with advanced chips, which by the way remains a challenge. That's definitely a dependency that remains for many countries.
There is an effort to address sovereignty at that point of the stack as well. There are growing examples of that. The Trudeau government in Canada just announced a plan for AI sovereignty that invests something like C$2 billion in the technical infrastructure layer of the stack. It’s a little bit up in the air whether that money goes to commercial cloud providers or towards actually building out their own government-owned infrastructure. But there is a focus on that part of the stack as well.
Kevin Xu: Regarding Canadian sovereign AI, a lot of people on the internet made fun of Trudeau’s efforts for being kind of wimpy — they only put down CA$2 billion. Even Singapore's commitment was bigger.
But that validates a lot of the stuff that we're talking about. It's not about the money per se. The Canadian population is still probably less than 35 million or something like that. If you do it from an investment per capita perspective, it's probably more than enough to codify whatever the Canadian government thinks is the Canadian version of the truth, the Canadian version of grounding or culture or whatnot.
The Canadian example is a good example of a player interested in hedging their bets and being a part of the conversation.
Government leaders don't want to lose out on being part of the conversation. They don't want be excluded from the next Bletchley Park AI Safety Summit. Maybe they would even like to be a contender to host such a summit — I think South Korea is next in line. That is part of the kind of politicking around the world when it comes to AI that has very little to do with economic return.
Pablo Chavez: I've seen that criticism and it makes sense. There's a lot of comparisons. For example, there’s the Microsoft-G42 deal. That was a $1.5 billion investment by Microsoft just in that partnership, so three-fourths of the Canadian investment in its entire national infrastructure.
Having said that though, companies and governments dip their toe in the water. Not everything that becomes big starts big. Look at the federal budgets for any number of governments. Singapore has an annual budget of something like $112 billion. That's a lot of money. Taking even 1 or 2 percentage points off of that and deploying it for the development of infrastructure could make a dent.
Perhaps it's not sustainable long-term, but the amount of investment at this point is not a criticism that resonates too much for me.
Jordan Schneider: One thing that makes sovereign AI easier is not having to rely on closed models. The big news this month was Mark Zuckerberg's release of Llama-3, which for all intents and purposes seems to be roughly comparable with what GPT-4 and Claude are currently putting out on the market.
I just want to take a little brief detour to the media strategy that they did to launch this by doing an interview on the Dwarkesh Podcast, which I think should be inspirational to all the listeners out there. The "Dwarkesh Arc" is very instructive.
He went from literally being in college, bored during COVID, and just put in more work than any other podcaster. While some people would prepare for an hour or five hours, he would prepare for two weeks for each episode and it really showed. I am so happy to watch him land incredible interviews like this. It just goes to show that there's always room for a new podcast.
Listeners out there, if you have the interest and intestinal fortitude do as much work as Dwarkesh, and you want to do something related to China or emerging tech, I am happy to have a chat and help you get off your feet. There is a ton of room, particularly in the China-focused space, to expand the global podcast offerings.
With that, let's talk a little bit about Llama. Kevin, what are your big takeaways of the model's release and Facebook's broader strategy of being the open player when it comes to AI? Open-er, I guess.
Kevin Xu: There is a bigger picture to what Zuck and Meta are doing when it comes to open source in general, even beyond Llama. For those of you who track the company closely, not just on the AI front, you probably saw that Zuck is also opening up the operating system of its virtual reality headset, Quest. They call it Horizon OS.
This is a very Android-esque strategy towards Apple's headsets or virtual reality product, which we all assume will be a closed system, just like iOS. Combine that with Llama's open source strategy from the get-go. Combine that even further with this other thing called the Open Compute Project from probably close to 10 years ago at this point. Meta, back then Facebook, initiated it to open source data center design.
Pablo, you've definitely seen the market impact of that. They entirely gave away the blueprint for how to design the most efficient, the most cooled, and well-performing data center in the world. They did this to push back against the then monopoly power of AT&T and other telecoms, and probably Google, which also has its own powerful but proprietary data center.
There is a reason why there is actually a long history of Meta opening infrastructural technical capability that has nothing to do with their actual business model, which is selling ads. It's a very smart way to erode the marketplace moat of other competitors who have closed versions of the same thing.
Notice I did not talk about geopolitics. I did not talk about sovereign AI. I did not talk about any of this stuff that we like to talk about when it comes to open source. There's a very strong business strategic motivation to open sourcing all the stuff that you don't necessarily need to monetize directly.
So the Llama model open sourcing up to this point is a direct way to erode the moat of OpenAI's model, of Google's Gemini model which is mostly closed, and some of these other competitors that Meta is trying to battle in the marketplace.
Do open source models have some unintended consequences when it comes to giving our competitors, or our adversaries, enough technology for them to either catch up or to do some bad things? Zuck actually had some kind of hedgy answer to those questions when Dwarkesh asked him about bioweapons and stuff like that in the podcast, which as Jordan mentioned I think is very much worth listening to.
It's still very much a TBD answer in Zuck's mind when it comes to how dangerous it is to open source. There is a lot of innovation and a lot of broader knowledge exchange benefits to open sourcing almost as much as you can, when it comes to technology.
Pablo Chavez: From a nation-state perspective, it's very interesting to see that a lot of the sovereign AI strategies have been built on open source technology and have deployed open source technology. If you look at the Falcon LLM in the UAE, that has been an open source model. France has been very focused on supporting and deploying open source models.
There's a part of me that thinks that these national strategies are actually very akin to what Kevin is talking about in terms of Meta's strategic plan. Meta essentially benefits from deploying open source AI because it closes the gap between itself and its competitors. It also attracts developers to its ecosystem and has a lot of positive brand impacts.
A number of these countries are thinking about the same thing. If you use and deploy open source AI, perhaps you narrow the gap between the rest of the world and the US and China, who have a strategic advantage.
Going back to the UAE model, one of the things that they were talking about very much was open sourcing in order to attract developers and to attract technologists to its ecosystem. That includes physically moving to the UAE for it to become an AI hub. So that is a positive outcome.
From a branding perspective, the open models have an appeal that has been to Meta's benefit and perhaps will also be to the benefit of countries like France.
I do agree that there is a national-security question that has not been answered yet by proponents of open source. There is this kind of catastrophic risk that's out there with open source AI.
There hasn't really been a good answer. Perhaps there doesn't need to be a very good answer right now. Perhaps now is the moment to diffuse and democratize the technology and maybe at some future point, you start to put guardrails and limits on it. That's essentially what the Biden administration has suggested with its executive order.
Open source is definitely a key component to all this. It's very much a trend to watch, as Kevin has suggested.
Kevin Xu: I want to underline what Pablo just said. It’s the talent dimension. I've been working in the open source industry for a pretty long time at this point. It's certainly a debatable topic still, whether open source or commercial open source is a good go-to-market model or not, but no one disputes the fact that open source is a great talent acquisition model.
That's another reason why Meta has open sourced so much of its underlying infrastructure. Google has been a huge contributor to that in so many fronts when it comes to big data or container orchestration software. Countries are coming around to the fact that this is a huge way to attract talent from around the world.
We talk about how sovereign AI actually needs talent. You can't just buy the software or the hardware or have the energy. You still need people to build the technology, run and maintain it, and build on top of it for years to come. So the open source talent acquisition nexus has been well recognized in the sovereign AI conversation.
Open Source Risks and Governance
I want to finish with another point, on the scary dimension of open source. I want to push back on all the folks who are listening who think open source is scary, and that it's going to destroy the world and give all of our adversaries secrets to weapons and whatnot. Open source is not a recent phenomenon.
It's been around for three or four decades at this point. One of the most prominent pieces of software, Linux, which is the root operating system of so many different machines, is open source. That piece of software being open, and usable to all countries and all players, hasn't become the reason why we have non-stop cyber attacks that steal secrets and data from around the world. That’s a good example there.
Having infrastructure critical software out there in the open actually makes software and technology more secure, not less secure, because of the number of eyes around the world that are always ready to patch up any security vulnerability. Proprietary, within-the-closed-wall software development is actually more vulnerable.
So that is a bit of a soapbox that I have. I'm obviously happy to listen to the other side and be corrected as always. That's where I'm coming from when it comes to open source being a feared aspect. It's vastly overblown and not necessarily supported by history or evidence.
Pablo Chavez: Your point is very valid in terms of open source software itself being secure because there's a community. For example, around Linux they patch vulnerabilities left and right. That's a fairly proven model. The issue with AI of course is not necessarily that the AI itself would be vulnerable, but rather that it can be deployed for terrorism or bio, you name it. The parade of horribles is out there.
The open source community has not quite come up with a particularly compelling answer to that question about misuse and what you do about it. Perhaps the answer is just that it’s part of the risk in the global deployment of advanced technology. But it is one of the remaining questions.
There’s one point that's been made to me. What if we were talking about advanced chips that were open source? Would that be a good thing? Aren't we benefiting from the fact that that’s proprietary technology? At the very least it helps the US government slow down the PLA's technological advancement. That’s just an example.
Kevin Xu: There’s one helpful way that I've been thinking about this. It’s a very important topic, so I don't want to dismiss the fact that there aren't risks out there. There is the "open sourcing stuff you can touch" category. Then there's the "open sourcing stuff you cannot touch" category.
We can put hardware, GPU, RISC-V related stuff in that world. There is probably a good way to think about how to properly restrict the degree to which you open source some of this know-how so we still glean or accrue the benefit of innovation globally, without giving away the secrets that could harm us. That's the stuff you can't touch.
When it comes to pure digital code, open source software, there's a very good set of historical examples to show how open sourcing code will only make the code stronger and more secure. If you're afraid that that code can be applied and abused for bioweapons, nuclear weapons, terrorism acts, etc., those are physical products and activities that are harmful.
The right way to think about regulating them is to still regulate the actual materials that could be used to make chemical weapons or bioweapons or nuclear weapons. We still need to have a regime around restricting uranium misuse, not the code that could possibly lead to them being made in a better way. That's an easier way to regulate.
Jordan Schneider: That's tricky, right? With cyberattacks, you can't regulate the code out of it. You're just hoping that Facebook has done a good enough job beforehand to make sure that Llama can't be misused and repurposed.
Not to pick on open source too much, but we've got Kevin on the show today. There was a really nasty hack a month or two ago from what was presumably the Chinese state. It was a human engineering one. Basically, there was some critical node in the global internet infrastructure, that had one guy maintaining it and he needed help. Someone out there was being very obsequious and polite and helpful until they weren't and put a backdoor in half the internet. Yes, it eventually got caught.
Still, I think there are challenges that are inherent in the open source ecosystem. You just have nice Brazilian coders who are doing stuff out of the goodness of their heart that the entire world ends up relying on. You end up having maybe a different set of vulnerabilities than you would using closed-source software.
Kevin Xu: That's right. As the open source proponent, I'm always happy to be on the show to be the straw man. On that kind of hack alone, I need to look more into the details. I'm aware of the basic facts. That is a bigger problem of why do we have these one-person, part-time maintained packages out there that are so critical to so many different technologies?
Then there’s the fact that it’s still being used because it is open source. If the solution is to close that, then we need a company or a government or an institute that will need to fund the ongoing maintenance and development of that on an ongoing basis.
I don't know if that's a closed or an open question versus a "where should we actually fund our infrastructure critical software" question.
Pablo Chavez: That's a really good point. To Kevin's point, I don't know that it's open versus closed necessarily in this context. It is a question of responsibility. There are plenty of examples of closed source software that has been very vulnerable and has been very exploited over the course of the last several years. So we could probably go chapter and verse through both sides and find vulnerabilities on both sides.
On the open source and AI sovereignty point though, it's just fascinating to see the leveraging of open source by countries and the deployment of open source models by countries. AI sovereignty is sometimes a little bit of a misnomer because these countries are doing a little bit of AI sovereignty jiu-jitsu.
They're actually putting this in the context of AI sovereignty, but ultimately it's about deploying the technology more broadly, diffusing it, and democratizing it with a strategic goal of narrowing the gap. It's just been interesting to see how the open source part of this has played out in this effort towards AI sovereignty.
Jordan Schneider: The big question is going to come in 2025 and 2026 when there's only Anthropic, OpenAI and Facebook left. Let’s say we're in a world where China can't develop leading-edge models on their own because of hardware restrictions or talent issues. We’ll see if the US government allows Llama-5 to be released.
It might immediately upgrade all of China's domestic, commercial, and military capabilities because it’s basically a few years’ worth of fast-forwarding. There are going to be more serious catastrophic risk and national-security competition questions that will be asked of these models.
What’s China’s plan to develop sovereign AI in such a circumstance?
Kevin Xu: Whether they use the term or not, China has recognized that sovereign AI is a very important dimension or vector of the “New productive forces” 新质生产力 agenda that the party is pushing.
You can see a lot of vectors where generative AI in the consumer sense may be restricted. I have a whole soliloquy on that that I'll save for another day.
When it comes to using elements of generative AI to boost the productivity of advanced manufacturing as the country ages — or boosting self-driving, which is still very prominent in China as a vector of development — we will probably see a lot more of that get implemented. That's the desire from the government at least, to get that implemented to boost the productivity of the country overall in the next stage of economic development.
There is a very good chance that could happen, if only because so much advanced manufacturing does exist physically in China. Some people might call that overcapacity. In my mind, that means there are lots of use cases to train your robots. It’s like a volume game when it comes to training the best robotic model or the robotic arm to make the next thing.
You need those scenarios on an ongoing basis to improve the accuracy and ability of that particular application of AI, which I think China has some advantages in. We'll see whether they're able to reap that benefit in the next few years or so.
The TikTok Ban
Jordan Schneider: Let's close on TikTok. Pablo, as a former government relations professional, what takeaways do you have from the way this ended up playing out?
Pablo Chavez: Look, last year I wrote an article for Lawfare about how regardless of the outcome, it would be important for Congress to weigh in. I'm glad that Congress did weigh in.
Ultimately the decision is fine in a lot of ways, but it's actually quite complicated. That complication worries me. So at a high level, whether you like to characterize it as a ban of an app or not, effectively at some point it may be a ban, if TikTok isn't sold.
With that in mind, it looks like the US is joining the app-banning club that does not have great members. It's China, Iran, and some other players.
Jordan Schneider: And India.
Pablo Chavez: Well India is going through a bit of a democratic recession. So perhaps it’s not the country to be compared to at this time on this topic.
But if you scratch the surface a little bit, some of the free-expression concerns fall away. To my mind, it should be fairly easy for TikTokers to move to other platforms. They're not going to be censored on those other platforms.
So I have some doubts about the strict “free expression” arguments. However, the US government is going to have quite a challenge internationally because it is probable that a number of countries are going to use the TikTok example to take action against American social networks.
To that end, it's going to be super important for the State Department and other parts of the federal government to distinguish very clearly between the action that the US government is taking and app bans in countries like China.
The process is going to have to be emphasized. It’s actually really important. It was a rule-bound, transparent, open process that led to this legislation. Again, it isn’t going to have much of an impact on the free expression rights of individual Americans. So that has to be emphasized. The system has to be emphasized as well. This is the exception to the rule versus the rule in countries like China.
Finally, there’s scope. We're going to have to be very clear about the fact that this was about restricting an app that, if the reports are correct, is effectively controlled by a foreign adversary country that may be interested in interfering with our democratic system.
Some of the harder parts of all this are coming. It's going to be a heavy lift for the US government vis-a-vis other governments around the world.
Jordan Schneider: My big hypothetical is to what extent it was them playing this hand incredibly poorly that led to the legislation getting passed, versus just a difficult fact pattern for TikTok to argue its way through.
Here’s my reading of the history of the past four years. Let’s say they had taken Project Texas more seriously. Let’s say they had actually started to set up real walls between TikTok and the China corporation. Let’s say they had done a little better and been more contrite in the hearing. Let’s say they didn't take their foot off the gas when it came to this thing and hadn’t been super overconfident.
Let’s say they had maybe not even done the Uber-style "save TikTok" notification push just a few months ago. Let’s say Speaker Mike Johnson didn't come to Jesus when it came to funding the aid bill and didn't have this weird Republican split that he had to mollify with the TikTok ban of all things. Then TikTok could have wiggled its way out of it, even though there are all the legitimate questions that we've been talking about on ChinaTalk for the past five years.
Pablo, to what extent do you think the fact that this legislation actually happened is contingent as opposed to inevitable?
Pablo Chavez: You're right that a lot of things had to go wrong for TikTok for this to pass. Having said that, this is where the trend is going. Look across the board. Look at what USTR did in pulling away from its advocacy against data flow restrictions, data residency requirements, and so forth.
Going back to the AI sovereignty conversation, this seems to be where a lot of internet governance is going.
Looking at the TikTok case through the lens of sovereign AI, TikTok is ultimately driven by an AI algorithm. It may be an early example of an AI product being blocked from a country.
The reason that the US can do it is because we have the ability to replace that product with homegrown products that are as good, if not better, than what TikTok was providing. That probably softens the blow and it gives us a bit of a sense of what the geopolitical future is going to be in the digital space on this stuff.
Sovereign AI Cyphers
Jordan Schneider: At the end of 2022, I did a show with Dylan Patel and Doug O'Laughlin and we were so excited with ChatGPT that we ended up writing a semiconductor rap. I had to read it out myself. Here we are in spring 2024 and we now have apps that not only write lyrics, but also produce entire songs for you.
So what we're doing today is a little bit of a sovereign AI TikTok topical battle, with Kevin, Pablo and I each bringing one song to the floor. Pablo, make your pick for your contestant.
Pablo Chavez: I called mine "Sovereign AI ‘24," and it's essentially a song about Sovereign AI in the style of 80s synth pop. I thought it was really great.
Just one last thing. I thought about the outcome of sovereign AI and what it mean for the future. There are basically two views of sovereign AI. The first is the Vinod Khosla view. This is good. This is the democratization of this technology. It gives countries some amount of control, which is a good thing. It also promotes diversity, which is a good thing.
Then there's the Ian Hogarth view. This is actually an arms race that's being facilitated by these efforts at sovereign AI. We don't know but those are the two potential scenarios. It's still not clear to me whether the general outlook is "hey, cars are coming, so we're going to build roads," or "war is coming, and we've got to build rockets." Where we end up in that debate is up in the air right now.
That's one thing that's reflected in my song as well.
Kevin Xu: My AI-generated song talks about sovereign AI and why Nvidia is going to take over the world.
Jordan Schneider: Awesome. Mine is about the TikTok ban.
