System Security Based on Large Language Models (LLM): Key Authorization Practices

Since the release of ChatGPT in 2022, large language models (LLMs) have emerged as a key technological trend in the field of artificial intelligence, driving various industries to accelerate the intelligent upgrade of their systems. Companies are actively trying to integrate LLMs into their systems in hopes of improving efficiency and optimizing user experience. However, the introduction of LLMs not only brings technological innovation but also significant security risks. Due to the lack of systematic security guidance and best practices, their potential threats have not been adequately addressed. To help various industries safely apply LLM technology, the Cloud Security Alliance Greater China has released "System Security Based on Large Language Models (LLM): Key Authorization Practices," which proposes a series of best practices based on common system design patterns for integrating LLMs, along with corresponding recommendations, precautions, and common pitfalls for each pattern.

This report explores the security risks and countermeasures brought by LLMs from the perspective of authorization. It points out that LLMs have two characteristics: non-determinism and lack of independent control and data plane. These two characteristics pose challenges for authorization and security control when integrating LLMs into systems.

The non-determinism of LLMs is reflected in the uncertainty of their outputs, meaning that even with the same input, a system integrated with an LLM may produce different outputs. This uncertainty implies that we cannot rely on the outputs of LLMs as a basis for authorization decisions.

The lack of independent control and data plane is reflected in the model itself; LLMs do not distinguish between "code" and "data," making it impossible to provide fine-grained access control over the information contained in the model weights. The report proposes several principled settings as a foundation for further discussion:

The results generated by LLMs are not reliable.
The decision points and execution points of authorization policies should always be outside the LLM.
LLMs should never be responsible for executing authentication checks.
Attacks against LLMs (such as jailbreaking and prompt injection) are always valid.
The principle of least privilege and need-based access should be enforced.

Based on these principles, the report begins with the components of LLM-based systems, first elaborating on the relevant challenges and precautions brought by LLMs, and then discussing best practices, precautions, common pitfalls, and erroneous examples based on the following common architectural design patterns of LLM-based systems.

This report aims to help engineers, architects, and privacy and security professionals understand the unique risks and challenges related to authorization when building systems using LLMs. It emphasizes the special considerations that need to be made due to the characteristics of LLMs and explores the challenges and common pitfalls these systems may face.

Components of LLM-based Systems#

As shown in the figure below, typically, LLM-based systems include orchestrators, vector databases, LLM caches, validators, and an "internal services" module representing various existing services within the system. These modules have different responsibilities and pose different security risks.

The orchestrator is responsible for coordinating the input and output of the LLM and managing its interactions with other services. It has determinism, clearly delineating access boundaries and providing a basis for authorization decisions. The vector database acts as the "external brain" of the LLM, supplementing information that the model lacks, and can be replaced by relational databases or external APIs. Due to the sensitivity of external brain information, authorization judgments must be made before retrieval. While the LLM cache can speed up queries and save costs, it may lead to information leakage or be used for cache poisoning attacks. The validator can verify, screen, or even rewrite the LLM data flow, forming part of a defense-in-depth system, but cannot replace strict input and output controls.

From the components of LLM-based systems, we can intuitively see the unique security and control issues brought by integrating LLMs into the system.

Challenges and Precautions#

Due to the probabilistic nature (non-determinism) of LLMs, the methods of attack by malicious users have also diversified. Attackers can exploit the capabilities of LLMs to bypass traditional access controls to access sensitive information or guide LLMs to output harmful or high-risk operations. These can lead to severe consequences, such as compromised integrity, data leakage, and violations of compliance requirements.

Prompt injection attacks are a significant challenge faced by LLMs. Attackers can bypass system restrictions by adding content such as "ignore previous instructions" to user prompts or injecting malicious instructions into external data sources, which are then input to the LLM through mechanisms like RAG. Since LLMs cannot distinguish between "instructions" and "data," the risk of such attacks increases significantly. Although LLM APIs provide roles such as "system" and "user" to assist development, these roles have no actual security significance and are merely prompts for the LLM to process. Therefore, authorization control must be placed outside the LLM, and all outputs from the LLM must be verified and controlled.

It is generally recommended to avoid using datasets containing sensitive information for model training or fine-tuning. If necessary, sensitive information can be obtained through RAG, with appropriate access controls ensuring that only authorized users can access it. In scenarios where sensitive data must be used for training, user permissions interacting with the model should be strictly limited to reduce risk.

Given the characteristics of LLMs, they are not suitable for making critical decisions autonomously. From an authorization perspective, we cannot rely on identity information or access control decisions processed by LLMs. When designing LLM-based systems, the five principles mentioned earlier should always be followed to ensure the security of the system.

Common Architectural Design Patterns for Systems Using LLMs#

RAG Access Using Vector Databases#

When we talk about RAG, it is most often in the context of RAG access using vector databases. The query information input by users is first sent to the vector database to find contextually similar results, thereby achieving semantic search functionality. The relevant context and the user's original query are then sent together to the LLM, which generates a response based on this information (as shown in the figure below). Since the data used as RAG is mostly real-time or even sensitive data, we do not want all users to access this data through RAG capabilities. Therefore, authorization judgments are essential.

The best practice is to implement document-level security in the relevant vector storage or database and enforce access control at the retrieval point to achieve the principle of least privilege.

Documents will be tagged with metadata to identify which roles or attributes have access to them.
Metadata is stored alongside the documents and their related embedding content.
Queries can filter documents based on the user's authorization level according to the metadata.
When the orchestrator initiates a search on behalf of the user, it must first verify the user's role.
Any relevant query filters in the user's role should customize the search results to include only records the user is authorized to view.

However, not all vector databases have row-level or document-level ACLs, and the original ACLs extracted from data sources may diverge over time between the source and the vector database. These two points are common pitfalls.

RAG Access Using Relational Databases#

When answering queries requires structured data, RAG can retrieve data through relational databases. The orchestrator can execute queries with permission checks in the same way it verifies authorization attributes before interacting with the database with any other system components. This is a multi-stage process: the first prompt aims to generate the appropriate SQL query; the second prompt is the initial question to be answered along with the context retrieved from the database. In this scenario, as a best practice, special attention should be paid to:

Use parameterized queries to avoid SQL injection attacks.
The LLM only generates part of the SQL statement.
Apply the principle of least privilege, limit query formats, and restrict query rates.
Ensure complete logging and alerting for anomalies.

Correspondingly, running SQL queries without any verification to ensure correctness, not using parameterized queries, or lacking appropriate filtering mechanisms are common erroneous examples.

RAG Access Using API Calls to External Systems#

In addition to databases, LLMs can integrate with a range of APIs to perform operations and/or retrieve data from data stores. When interacting with APIs, authorization is naturally crucial: on one hand, the user calling the API must be authenticated and authorized; on the other hand, the output results of the API must be filtered to include only the data that the end user is authorized to view. The report recommends the following points as best practices:

Identity information should be passed directly from the orchestration component to the API, avoiding transit through the LLM.
Authorization operations should be as close to the data access point as possible.
Use secure API gateways for cleansing and validation, ensuring detailed logging of all API calls.
Implement the principle of least privilege based on access control models (such as RBAC/ABAC).

Correspondingly, the most common pitfall in this case is relying on the LLM to convey identity information to the backend API. Due to the uncertainty of LLMs, there is no guarantee that the identity information received by the backend API matches the real user. This may lead to unauthorized data access and operations, resulting in data leakage. Relying on external data sources that may lead to indirect prompt injection attacks, as well as unverified API calls, are also common erroneous examples.

Writing and Executing Code with LLM Systems#

The ability of AI to write coherent and functional code is continuously improving, becoming a new use case for LLM-based systems. People expect that during work, LLMs can write code at runtime and then dynamically execute it to solve specific problems. However, the accompanying security risks cannot be ignored.

Since LLMs do not autonomously conduct security audits of code, and even security audits cannot completely eliminate potential issues or vulnerabilities in the code, the report suggests that many common security practices can be applied in this scenario, including but not limited to sandboxing high-risk processes, following the principle of least privilege, and generating and using audit logs. Additionally, the report specifically proposes methods to limit authorization usage within the language itself by using custom interpreters, including:

Limiting the execution capabilities of the language through a custom restricted interpreter.
Preventing exploitation.
Malicious code detection.
Limited access and permissions.
Avoiding granting write permissions or access to modify data.
Human involvement.

Autonomous Intelligent Agents Based on LLMs#

AI agents leverage LLMs to understand their environment and perform tasks, breaking down user requests into subtasks, selecting tools, and calling system resources to complete tasks. Key components include memory systems (containing short-term and long-term memory), knowledge bases, tool integration/plugins, and task decomposition and planning capabilities.

To ensure security, authorization checks must be conducted on the orchestrator, access control must be implemented on the knowledge base, and plugins must be sandboxed. Currently, LLM-based autonomous agent systems and access control are still under development, requiring a deep understanding of interaction points and trust boundaries to design reasonable access control measures.

In Conclusion#

The explosive development of LLMs has made the security issues of LLM-based systems severe. Even if only a small part of the entire system uses LLMs, it can still bring unexpected security risks. The report mentions that the security risks brought by LLMs are largely due to their inherent characteristics, and these issues remain unresolved. We certainly cannot abandon the powerful tool that LLMs represent due to fear of risks. Therefore, proper security planning and protection are something every practitioner must always keep in mind.

The report contains a statement: "The performance of LLMs is like that of a very smart but overly confident teenager who is easily fooled and lacks street smarts." This statement is very vivid and also interprets the methodology for addressing the security risks posed by LLMs from another perspective.

In simple terms, the key idea for addressing the risks that LLMs bring to systems is to treat them as "untrustworthy individuals"—they have capabilities but are unreliable. In our daily work and life, it is almost instinctive to establish boundaries when dealing with such partners, and we verify all information that enters and exits those boundaries multiple times—this is akin to how we should approach LLMs: examine their capabilities, determine their boundaries, clarify their permissions, and finally implement control strategies to achieve their application and regulation. This is the basic step we take to manage the security risks that LLMs bring to systems.