What DeepSeek Tells Us About Cyber Risk and Large Language Models

Not many companies can single-handedly wipe out $1 trillion in market capitalisation from the US stock market. Yet that’s exactly what Chinese AI startup DeepSeek managed at the end of January after introducing a new model that is claimed to operate at a fraction of OpenAI’s cost with similar results. Since then, markets have recovered, and news of serious security and privacy issues with the DeepSeek-R1 large language model (LLM) and the firm’s front-end app has emerged.

But before CISOs shrug their shoulders and move on, let’s put this in context. Just because DeepSeek’s technology has been labelled high risk does not mean other models are completely without fault. Security teams may need best practice standards to help them navigate risk in this fast-evolving space.

What’s Wrong with DeepSeek?

According to one piece of research, DeepSeek-R1 has two main issues:

• It is vulnerable to “jailbreaking” via prompt injection. In other words, by entering specific prompts, users can bypass the built-in safety guardrails inserted by DeepSeek developers – resulting in some unethical and downright dangerous outputs. For example, when Kela researchers prompted the LLM to adopt an “evil” persona, free from ethical or safety constraints, it was quite happy providing an infostealer malware script, suggestions on which cybercrime marketplaces to visit, and even guidance on creating a suicide drone
• It is prone to “hallucinations” – for example, providing when prompted a list of personal information on senior OpenAI employees, which was false

A separate study from EnkryptAI confirms that DeepSeek is prone to delivering misinformation and harmful content. It claims that the model is:

• 3x more biased than Claude-3 Opus
• 4x more vulnerable to generating insecure code than OpenAI’s O1
• 4x more toxic than GPT-4o
• 11x more likely to generate harmful output versus OpenAI O1
• 3.5x more likely to produce Chemical, Biological, Radiological, and Nuclear (CBRN) content​ than OpenAI O1 and Claude-3 Opus

Additional concerns over the security of DeepSeek’s back-end infrastructure emerged after one security vendor discovered a publicly accessible database belonging to the company, exposing highly sensitive data, including log streams, API secrets, and operational details.

Separate analysis from SecurityScorecard reveals a raft of security and privacy issues with the DeepSeek Android app, including:

• Weak security, such as hardcoded encryption keys, weak cryptographic algorithms, and SQL injection risks
• Overly broad data collection on users, including inputs, device details, and keystroke patterns, all of which are stored on servers in China
• Undisclosed data sharing with Chinese state-owned enterprises and TikTok parent ByteDance, and vague privacy policies

Anti-debugging techniques which are usually deployed to obstruct security analysis.

Lifting the Lid on LLM Risks

However, while rival models like OpenAI’s are thought to be far more secure, it would be foolish to assume that the risks highlighted by DeepSeek-R1 are not present elsewhere.

“The unfolding DeepSeek incident should not be exploited as a convenient reason to suddenly forget about serious violations and AI-related risks posed by other GenAI vendors. Otherwise, we are missing the forest for the trees,” argues ImmuniWeb CEO, Platt Law cybersecurity partner and Capitol Technology University professor Ilia Kolochenko.

Whether organisations are using a third-party LLM like DeepSeek or developing/fine-tuning one in-house, they must be aware of how it can expand the corporate attack surface. Potential points of risk include the model itself, the data it’s trained on, any APIs, third-party open-source libraries, front-end applications, and back-end cloud infrastructure.

OWASP has compiled a Top 10 for LLM Applications listing the main security issues – some of which impacted DeepSeek. These are:

1. Prompt injection vulnerabilities which can be exploited by crafting specific inputs to alter the model’s behaviour, bypassing safety features.
2. Sensitive information disclosure which may include corporate secrets or customer data.
3. Supply chain vulnerabilities, such as bugs in open-source components, which could be exploited to create unintended outputs, steal sensitive data or cause system failure.
4. Data and model poisoning, where pre-training, fine-tuning, or embedding data is manipulated to introduce vulnerabilities, backdoors or biases.
5. Improper output handling, resulting from insufficient validation, sanitisation and handling, potentially leading to hallucinations or introducing security vulnerabilities.
6. Excessive agency stems from excessive functionality, permissions, and/or autonomy and could lead to a host of negative outcomes, including breaches and compliance issues.
7. System prompt leakage, which occurs when system prompts contain sensitive information, allowing attackers to weaponise this insight.
8. Vector and embedding weaknesses are specific to LLM systems using Retrieval Augmented Generation (RAG) and could be exploited to inject harmful content, manipulate model outputs, or access sensitive information.
9. Misinformation, which stems largely from hallucinations.
10. Unbounded consumption, which stems from “excessive and uncontrolled inferences” and could lead to denial of service.

Can ISO 42001 Help?

The good news is that CISOs looking to harness the power of LLMs within their operations and/or to deliver to customers can do so in a way that mitigates these risks, thanks to a groundbreaking new standard. ISO 42001 provides a framework to establish, implement, maintain and continually improve an AI Management System (AIMS). Covering the entire lifecycle of AI systems, it helps organisations to:

• Embed ethical principles into the AI to avoid bias and respect human rights
• Increase the transparency of AI systems and algorithms in order to drive trust and accountability
• Identify, assess, and mitigate risks like those highlighted by OWASP and found in DeepSeek
• Enhance compliance by aligning AI operations with existing legal and regulatory frameworks
• Foster a culture of continuous improvement in AI system management

Kolochenko tells ISMS.online that such standards aren’t a panacea but can serve a valuable purpose.

“The novel ISO 42001 standard will certainly bring value to fill out the regulatory vacuum in the realm of AI, although incidents involving AI – including very serious ones – will likely continue to grow exponentially,” he argues.

Corian Kennedy, SecurityScorecard senior threat insights & attribution manager, goes further.

“Both ISO 42001 and ISO 27001 provide governance and security frameworks that help mitigate risks from insecure third-party apps like DeepSeek and high-risk LLMs – whether external or internally built,” he tells ISMS.online.

“Together, they help to reduce risks from insecure AI models by enforcing strict governance, strengthen regulatory compliance to prevent unauthorised data exposure, and secure internal AI systems with access controls, encryption, and vendor due diligence.”

However, Kennedy points out that while ISO 42001 can provide a “solid foundation for AI security, privacy and governance”, it may lack contextual business risk.

“Therefore, it is the responsibility of those in cyber defence to implement additional controls based on the context of the threat landscape and in support of the business,” he argues.