Singapore is deploying AI at speed – and security risks are catching up

A misconfigured agent, flawed prompt, or manipulated instruction can significantly disrupt business operations.

Last April, K. Shanmugam, Coordinating Minister for National Security and Minister for Home Affairs set out a clear ambition: Build sovereign AI capabilities that create leaner government operations.

Initiatives such as Phoenix, a set of AI models built by Singapore’s Home Team Science and Technology Agency (HTX) with Mistral AI and Microsoft, signal a shift from exploring to operationalising AI.

AI presents a strategic opportunity but also introduces a new class of risk that traditional security models are not designed to handle. As Singapore accelerates AI deployments across government bodies, building the capability to maintain control, trust and accountability at machine speed is critical.

Singapore has built a global reputation for trusted digital infrastructure. From Smart Nation initiatives to its leadership in digital governance, the country has consistently paired innovation with discipline.

But AI models and agents, as they make decisions and act at a speed and scale that far exceeds human workflows, introduce a new risk landscape that calls for a radical shift in cybersecurity posture. In a public sector environment, agents might interact with citizen data, operate across agencies, and play a role in essential processes in real time.

We are starting to witness how a misconfigured agent, a flawed prompt, or a manipulated instruction, can have catastrophic ramifications on the security of sensitive data, and significantly disrupt business operations.

The first way to prevent this kind of scenario is building visibility.

Many organisations cannot clearly identify where AI models and agents are deployed or what they can access. As government agencies cater for the acceleration of AI usage among their teams, they need to ensure they maintain comprehensive visibility across all AI experiments and deployments.

They also need to ensure security is built in the fabric of AI deployments.  

Agents are developed and rolled out at high speed, and existing governance models, which are built around periodic reviews and human checkpoints, are too slow for this era. Governance needs to operate at the same speed as the systems it oversees, with continuous monitoring, real-time policy enforcement and the ability to detect and respond to unexpected behaviour instantly.

Access controls, which are usually defined around roles, must also evolve.

Agents are not tied to specific roles and are likely to operate dynamically across tasks and systems based on instructions. A single agent may analyse data, process workflows across different departments, and interact with multiple platforms within minutes.

In an agentic environment, trust is continuously verified, not assumed, and permissions must evolve dynamically, and in real time, based on the agent’s operations. As we have learned with zero trust, access should be granted only when required for a specific task and removed immediately after.

AI agents also force a rethink of traditional incident response models. Currently built around human actions, they now need to consider incidents related to autonomous agents, which can range from an agent producing wrong outputs, being manipulated through malicious instructions, or quietly overstepping its intended boundaries in ways that go undetected until the real damage is done.

To address this, incident response teams within government have to develop a deeper understanding of agent behavior, and edit their playbooks to define the evidence that matters in an agentic investigation, including the instructions an agent was given, what it was allowed to access, and the decisions it made along the way.

Just as organisations have developed processes for compromised users and applications, they now need equivalent maturity for AI-driven operations.

Red teams, who are responsible for stress-testing systems and identifying weaknesses before they are operational, also need to sharpen their offensive tools to evaluate the security of AI agents. They not only need to address technical vulnerabilities, but also develop the skills to test how agents respond to unexpected scenarios, ambiguous instructions, attempts to manipulate their decision-making, and access requests that fall outside their intended purpose.

This shouldn’t be a set and forget process that occurs before agents are released. AI systems are continuously evolving, and their behaviour can change as models, data sources, integrations, objectives, and business processes are updated. Security validation therefore needs to become an ongoing discipline.

For teams involved in enabling AI throughout our country’s public sector and wanting to ensure that Singapore maintains its excellence in innovation and digital governance, these changes and best practices are simply non-negotiables. Beyond excellence, it is about building trust in an AI-driven world, which will be a key differentiator in the future.