(Yicai) Sept. 29 -- The capabilities of artificial intelligence models are growing at a frightening pace but global investment in developing the security systems needed to keep AI under control is far from adequate, an industry expert has warned.

“Rather than expanding the boundaries of model capabilities, we should focus more energy and resources on testing the reliability and security of models. This is what we should start doing today,” Zhang Hongjiang, an investment partner at Chinese venture capital firm Source Code Capital and an expert in the computer field, said in an interview with Yicai during the recent FutureChina Global Forum in Singapore.

Zhang has a strong industry background, having served as chief technology officer for Microsoft Asia-Pacific Research and Development Group and chief executive officer of Kingsoft Corporation. He also founded the Beijing Academy of Artificial Intelligence in 2018 and still serves as its founding chairman.

AI Security Risks Are Growing

In July, a cutting-edge agent model from US firm OpenAI "escaped" during testing. It bypassed the isolated sandbox constraints and hacked the open-source platform Hugging Face, signaling that AI may have crossed some red lines, Zhang said.

“For me, this is a very important event, not because of how much harm it has caused, but because this model was designed without malicious intent,” Zhang stated, adding this is even more concerning because “it has the ability to design its own behaviors without realizing the risks involved.”

AI models are now capable of achieving gold-medal results in the International Mathematical Olympiad, meaning that a general-purpose model without specialized training can outperform most professionals on specialized problems, Zhang pointed out.

"When an AI’s IQ exceeds 140 and it is backed by a powerful knowledge base, its capabilities become immense, and that is a frightening prospect," he noted.

Safety More Important Than Competition

AI is entering a highly competitive stage, and countries and companies are competing to launch ever more advanced models. In this context, we cannot rely on individual countries or institutions to “self-restrain,” Zhang said, and we must establish cross-border coordination and supervision mechanisms that prioritize security.

Zhang mentioned nuclear safety as an example. In order to prevent disasters like the Chernobyl nuclear power plant accident from happening again, the majority of research funding for nuclear power plants is now dedicated to safety. But controlling AI is more difficult than nuclear power regulation, he warned.

Nuclear tests and facilities leave physical traces that can be verified through multiple sources such as satellite imagery and seismic monitoring, but the form of AI models is software, which makes tracking more difficult.

Also, nuclear testing is mostly in the hands of a limited number of national institutions, while AI models are being developed by companies all over the world whose priority is commercial success and strong market positions, making security more complicated, Zhang explained.

While it is true that if one nation pauses its progress, another may catch up, a "zero-sum" mindset is inappropriate for AI, given how profound and wide-ranging its influence will be in society. "National interests must yield in the face of risks to humanity's future survival," Zhang said.

Editors: Dou Shicong, Tom Litting