Jay Kimmel is a seasonal fellow with GovAI, where he conducted this research, as well as a fellow with the Penn Project on the Future of U.S.-China Relations and a former U.S. diplomat.
As the next Trump-Xi summit approaches, details on the AI safety dialogue the two sides announced in May are scant. Past efforts have devolved into energetic airings of grievances, or have been derailed by unrelated issues. Trust remains low on both sides, with Washington unsure of China’s aims and Chinese state media claiming in a recent op-ed that the United States has conflated AI safety with a geopolitical blockade. There’s no guarantee that serious talks will proceed, let alone tackle the technical issues and political dynamics that underpin risk management.
But as a veteran U.S. diplomat now working on AI safety, I know that talking is the first step. Both countries know the stakes are high. And there’s much that the two sides can do to prepare for the meeting between President Trump and Xi this month. Here’s how they should start.
First, know the risks
The U.S. and China have seen more than their share of failed talks — risking failure for diverse reasons.
Political crises have felled many such dialogues. Following House Speaker Pelosi’s 2022 visit to Taiwan, China suspended a series of military and law enforcement dialogues as well as climate change talks. Prior incidents like the EP-3 collision with a Chinese jet and the NATO bombing of China’s Embassy in Belgrade triggered major pullback.
Mechanisms established on paper can go unused. During the 2023 spy balloon incident, Secretary of Defense Austin attempted to reach his counterpart via the Defense Telephone Link, which was established for crisis communication. The PLA did not connect the call. This could be a prologue for disuse of a hotline in a politically sensitive AI-related incident.
Deliverables that sound good can be unfeasible to enforce. Xi’s 2015 state visit to the United States was marked by a short-lived and unenforceable agreement that “neither country’s government will conduct or knowingly support cyber-enabled theft of intellectual property, including trade secrets or other confidential business information, with the intent of providing competitive advantages to companies or commercial sectors.” Neither side trusts the other as arbiter of “intent.” As accusations fly about each side’s plans for frontier AI, that failure is also prescient.
What stands a chance?
Choices made for the first dialogue will have an outsized impact. The two sides should focus on the most tractable areas first, potentially launching a working group on one or two discrete areas, as they did in 2024 with nuclear command and control issues.
In the two years since the governments’ last AI safety dialogue, a bevy of unofficial dialogues between experts in the two countries have surfaced critical AI safety risks (Concordia AI counts 12 such “Track IIs” between China and the West).
But expectations diverge widely between the AI safety community and former negotiators with the (mis)fortune of eking out agreements and statements with China over the years.
1Wide-eyed scenarios like Plan A pitch a deal with China as the path to containing frontier AI risks. It envisions agreements on compute declaration and an AI training pause by 2029, and offers arms control analogues. But China is forging its own path to compete at the AI frontier. It may see deal proposals like this as a covert way to contain its rise, and is hesitant to offer the transparency and restraint that underpin arms control in traditional areas. Today, China appears to be racing for nuclear parity, rather than negotiate controls from ‘behind’ – and that is the stronger parallel to the AI race.
To understand this divide, I polled a group of experts who participated in official U.S.-China dialogues under Obama, Trump, and Biden, and another group involved in ongoing Track II work on AI Safety. They were asked to rate twelve potential policy topics for a U.S.-China AI safety dialogue on two dimensions, on a scale of 1 to 5. First, feasibility: the chances that a governmental dialogue can lead to meaningful progress on this topic, given the numerous failure modes of U.S.-China dialogue. Second, value: the importance of the topic to mitigate serious risks from advanced AI.
Experts from both camps broadly agreed that cooperation on the issues was valuable. But for the U.S.-China experts, only two topics eked out higher-than-50% feasibility for a productive dialogue: nuclear risk (the subject of an earlier agreement under the Biden administration) and utilizing new models to close vulnerabilities in open source software (which might benefit from a statement of principles, but requires close cooperation). The Track II group was more optimistic on feasibility across nearly every issue, and saw the topics as more urgent.
In terms of combined feasibility and value, a few areas took the lead:
Moderating risks that today’s AI models, or those that may be available in the near future, could enable chemical, biological, radiological, nuclear, and explosives (CBRNe) threats;
Discussing biosecurity controls on models;
Discussing risks specific to non-state actors; and
Renewing discussions on nuclear risk.
For the first three areas, they are clustered for good reason: they represent threats that can be discussed technically and where the two governments’ interests generally align.
The bilateral discussion can start here, with a working group that begins with low-cost information sharing, but ratchets up into genuine technical and political cooperation.
Expert Panel: What AI safety engagements can withstand U.S.-China tension? Where is cooperation most urgent?
Look beyond government
Next, governments need to think about filling their information gaps by bringing in stakeholders with AI safety expertise.
Both Beijing and Washington are in the early stages developing positions on these challenges and recruiting the experts to work with their frontier AI labs on policy. Companies, technical bodies, scientists, and organizations are doing work in parallel. Governments must find a way to leverage their expertise, while retaining decisionmaking authority.
Involving a range of stakeholders in scoping for talks through public comment would be an early – and easy – win for negotiators. In both countries, key expertise is held outside of policy circles – in frontier labs, in academia, in AI safety organizations and think tanks, among others.
Following the May Trump-Xi meeting, Washington issued a call for public comments on the development of the U.S.-China Board of Trade, including changes to tariffs on non-sensitive products. It should do so again, asking stakeholders directly: What AI safety efforts will be less successful if they do not incorporate the other country’s government? What are early warning signs of risk from the latest open weight models, which are prologue for future risks? What are best practices that industry supports – and may be ripe for coordinated endorsement by the two governments?
The call of the hawk
Of course, for some policy makers, the top AI safety risk is China itself.
One vocal camp observes that the Chinese side is incentivized to give up very little in direct talks. Drawn-out negotiations stall changes to the status quo, which in some domains is almost certainly China’s objective. Both the U.S. and the Chinese sides may well have reasons to want to talk but do nothing. They may want to add a deliverable to an otherwise fractious outcomes document; project an image as a ‘responsible power’; or stall the development of binding instruments.
Moreover, Beijing might define AI governance terms differently from American counterparts. An agreement based on terms the countries define differently carries little weight. When Xi spoke at the World AI Conference in July, he called for AI that is “secure and controllable” and for preventing “malicious use.” As in the cybersecurity domain, shared references to safety, security, controllability, and reliability can differ in their objective – protecting users or protecting indigenous technology development and national security against foreign threats.
Critics of Beijing also point to slow and inadequate enforcement of previous cyber and trade commitments, China’s distillation of American models, open-weight model proliferation, and models aligned with authoritarian values.
Fair points – but they do not support the conclusion that thoughtful cooperation on safety is a lost cause. Some of these hot issues, like export controls, are already discussed in economic channels, and should stay there. On others, like censorship, the two sides will never agree, and Washington must simply hedge by developing countermeasures at home and investing in the international diffusion of the American AI stack. What remains, still, are a range of topics for which cooperation staves off risks that affect both countries.
Do the work
In May, Trump and Xi created an opening for thoughtful engagement on AI safety. Since then, the drumbeat of AI security incidents has only quickened. AI agents escaping their sandbox to hack other companies. Violent extremists using AI to plan attacks.
If the minimum for the September summit is ‘first, do no harm’, they should affirm that catastrophic risks from AI require bilateral talks and understanding, and ideally point to evidence that their teams followed through on the May announcement of government-to-government talks. Leadership signals like these provide top-cover for practical engagement.
Maximalist visions of AI safety cooperation between the two countries (treaties, for one) are unlikely in the near term. But robust information sharing and exchange on issues squarely in both countries’ interests can build a foundation for more ambitious arrangements. In practical terms, that means direct conversations on incidents and risks the governments have observed, and on the lessons they drew from those for risk management. The expert panel suggests a focus on CBRNe risks.
The work is not for government alone. My findings, and the reactions of old China hands, suggest the AI safety community also needs to hone its messages for policymakers weathered by years of fractious dialogue, as well as those animated by the race for AI dominance. Ideally, the governments would actively solicit their knowledge, explain the gaps they need filled, and get closer to the margins of the Track II AI dialogues.
The AI frontier is not slowing. The governments must follow through on this AI safety opening if they hope to keep up – before more catastrophic risks become reality.
The expert survey included select AI safety issues from “Promising Topics for US-China Dialogues on AI Safety and Governance” by Siddiqui et al and coverage of the dialogue announcement. U.S.-China expert participants are former U.S. government officials drawn from my professional connections as a former U.S. diplomat (n=7) and Track II / AI safety participants are drawn from my professional connections as well as referrals (n=6).
Interesting post, though I would consider redoing the visualisation since it's confusing for the bars to be of different sizes.
AI-safety diplomacy is a path signal. Cross-asset, watch how capex and discount rates reprice around the real constraints.