California Proposes Stronger AI Safety Rules and Emergency Powers

California State Capitol building in Sacramento under a blue sky
The California State Capitol in Sacramento.

California officials have proposed a stronger state framework for frontier-AI safety, including baseline development standards, independent verification and emergency powers for severe incidents. The document is a set of recommendations to the governor and Legislature—not a law already in force.

The California Government Operations Agency published AI Safety in 2026 and Beyond on November 16. Governor Gavin Newsom’s office highlighted the recommendations on November 18.

What the California AI safety framework recommends

The report calls for enforceable baseline standards covering how frontier developers identify, test and reduce severe risks. It recommends independent testing or verification rather than relying only on companies’ internal assessments.

It also proposes expanding the incidents that developers must report. The authors argue that present rules may miss events involving loss of control, cyber capabilities or AI systems used to accelerate further AI research.

For the most serious situations, the report recommends verified shutdown protocols and state authority to order emergency action. It further discusses a power to cap recursive self-improvement—the use of AI systems to accelerate development of more capable AI—when specified risk conditions are met.

Recommendations are not current requirements

The distinction between policy proposal and enacted law is essential. The report asks lawmakers to strengthen California’s framework. Its proposed standards, shutdown authority and recursive-improvement limits will require legislation and detailed implementation before they become binding.

California already has AI laws. The report describes SB 53 as requiring certain frontier developers to maintain safety plans, report covered critical incidents and protect whistleblowers. It also points to SB 813 and AB 1405, which establish elements of an independent-verification ecosystem.

Those existing measures form the starting point; they should not be conflated with every new recommendation in the November report.

How independent verification could work

Independent assessment can reduce the conflict created when a developer both builds a system and decides whether it is safe. But independence depends on access, funding, publication rights and freedom from commercial pressure.

A verifier may need controlled access to models, logs and safeguards that cannot be released publicly. Policymakers must therefore balance meaningful inspection with cybersecurity, privacy and trade-secret protections. Accreditation standards are also necessary so that “independent” does not become a label available to any consultant.

Future & Fact previously examined OpenAI’s embedded assessments, where access and publication conditions determine how much confidence an outside review can support.

The hard questions are thresholds and authority

A shutdown power sounds clear in principle but is difficult to design. Lawmakers would need to specify who can order a shutdown, what evidence is required, how quickly a developer can appeal and how the state verifies that the measure was carried out.

The same is true for recursive self-improvement. Regulators need measurable thresholds that distinguish ordinary software development from a dangerous feedback loop. Vague definitions could be ineffective or could capture benign research.

The report’s value is that it turns broad concern into concrete governance questions. The next stage is legislative: converting recommendations into definitions, due-process protections, technical standards and enforcement resources that can operate under real incident pressure.

Readers should watch the bill text that follows, not assume that every item in the report will be enacted unchanged.

The Future & Fact newsletter

Fresh discoveries.
Delivered to your inbox.

Explore AI, science and space with clear explanations and stories grounded in primary sources.

Coming soon — signups are not open yet.

Read our Privacy Policy for information about how we handle personal data.