Week4-Threat Modelling

Week-4: Threat Modelling

Ali Jaddoa, ,

Ali.Jaddoa@roehampton.ac.uk

Date:


25/26
Week4-Threat Modelling

RECAP: Last Week RA/RM


25/26
Week4-Threat Modelling

RECAP: Risk Management Process

width:1OO% center


25/26
Week4-Threat Modelling

RECAP: Cyber Risk Model

  • Risk arises when a TA exploits a vulnerability to harm an asset.
  • Risk increases with:Asset value; Vulnerability severity; Threat strength
  • Threat = Motivation × Capacity
  • Likelihood = Threat × Vuln
  • Risk = Likelihood × Impact

25/26
Week4-Threat Modelling

RECAP: Risk Analysis

  • Practical risk analysis usually considers two factors to determine the level of each risk
    • Likelihood (frequency/probability) of each type of incident.
    • Impact resulting from each type of incident
      center
  1. Qualitative Analysis
  2. Quantitative Analysis

25/26
Week4-Threat Modelling

Today's session: Threat Modelling

  • Introduction to Threat Modelling Concepts and Methodologies.
  • Threat Modelling Frameworks and Practical Examples.
  • Lab Building a Threat Model for an application.

25/26
Week4-Threat Modelling

What is Threat Modelling?

A structured approach to identifying, assessing, and addressing security threats in a system or application.
width:1OO% center

  • The art and science of assessing whether your defences are sufficient to counter relevant threats.
  • Systematically examining critical questions to identify potential security issues, often without relying on tools, enabling teams to uncover vulnerabilities and strengthen defences.

25/26
Week4-Threat Modelling

width:1OO% center

  • In cybersecurity, we look at assets, possible threats, and the defences or counter-measures we can put in place.
  • Controls reduce risk

25/26
Week4-Threat Modelling

Q: How is different from Risk Assessment (RA)?

  • RA: A compliance activity that evaluates existing systems to measure and document security risks, focusing on what has already been built.
  • TM: A proactive process to identify potential threats and guide changes in current and future system designs to enhance security.
Aspect Threat Modelling Risk Assessment
Objective Identifies specific attack paths and methods on systems Evaluates likelihood and impact of risks on the organisation
Focus Targets potential vulnerabilities and attacker actions Considers broader impacts, including financial and operational risks
Outcome Informs targeted defences against specific threats Prioritises risks for overall organisational management

25/26
Week4-Threat Modelling

Important Questions:

  1. What are you building?
  2. What can go wrong?
  3. What are you going to do about it?
  4. Did you do an acceptable job at 1-3?

25/26
Week4-Threat Modelling

Back to Basics

width:1OO% center

  • Vulnerability - Threat - Attack
  • Exploit: a successful attack
  • (NEW) Trust Boundary: where the level of trust changes for data or code

25/26
Week4-Threat Modelling

Threats 121

  • Threats represent a potential danger to the security of one or more assets or components
  • Threats could be malicious, accidental, due to a natural event, an insider, an outsider.
  • Threats have certain sources (Social, Operational, Technical, Environments)
  • A single software choice can result in many threats.
  • Threats exist even if there are no vulnerabilities
    • No relaxing
    • Threats change with system changes
      • How can a change in software result fewer threats?

25/26
Week4-Threat Modelling

Why Threat Modelling?

Enables organisations to proactively identify and mitigate attack vectors, prioritising finite resources for maximum impact.

  • Resource Prioritisation: Focus on critical risks within limited resources.
  • Threat Identification: Recognise potential threat events.
  • Threat Characterisation: Understand attacker TTPs (Tactics, Techniques, and Procedures).
  • Control Measures: Implement effective defences.
  • Lateral Movement Prevention: Limit attackers’ ability to accessing key assets.
  • Avoiding Blind Spots: Ensure comprehensive threat visibility.

25/26
Week4-Threat Modelling

Key Steps in Threat Modelling

  1. Identify Threats: Recognise potential attack vectors (e.g., phishing, weak passwords).
  2. Characterise Threats: Understand attacker methods (e.g., brute-force login).
  3. Prioritise Risks: Focus on high-impact threats within resource limits.
  4. Implement Controls: Use multi-factor authentication, monitor logins.

Outcome: Reduced risk of breaches, stronger security posture.


25/26
Week4-Threat Modelling

Threat Modelling Methodologies:

  • Asset-Centric Approach

  • Attacker-Centric Approach

  • System-Centric Approach

Activity Take five to do some reseach for each one


25/26
Week4-Threat Modelling

Threats Modelling Stages: MS

width:1OO% center


25/26
Week4-Threat Modelling

Methodology

  • Based on OWASP (Open Worldwide Application Security Project)

    Step 1 Scope Definition

    Step 2 System Decomposition

    Step 3 Threat Identification

    Step 4 Attack Modelling

    OWASP: is an online community that produces freely available articles, methodologies, documentation, tools, and technologies in the fields of IoT, system software and web application security


25/26
Week4-Threat Modelling

Step 1 - Scope Definition

  • Task A: Gather Information

    • software design document (SDD), technical specification or any system-related documentation.

  • Task B: Demarcate Perimeter Boundary: to determine the scope for threat modelling.

    • Components that support the functioning and running of the system e.g. servers, databases, client workstations, hosts, switches, routers etc.

    • Components that support the cybersecurity of the system e.g. firewalls, IDS and IPS.


25/26
Week4-Threat Modelling

Step-2 System Decomposition

Breaking down a system into its different components.

  • Task-A: components which a potential attacker may be interested in.

    width:1OO% center


25/26
Week4-Threat Modelling

Step-2 System Decomposition: Cont'

  • Task B: Draw How Data Flows

    width:1OO% center

    width:1OO% center


25/26
Week4-Threat Modelling

Step-2 System Decomposition

Task C: Divide Out Trust Boundaries: identify the respective limits of access, as well as the required levels of authorisation e.g. trust levels, granted to subjects.

width:1OO% center


25/26
Week4-Threat Modelling

Step-3: Threat Identification

Involves identifying A. threat vectors and listing B. threat events;

Useful threat information:

  • Timely: Information should be received in a timely manner as information that is outdated is useless to Users;
  • Relevant: Information needs to be relevant to the context of the Users. For example, industrial control systems may have different priorities compared to financial institutions; and
  • Actionable: Information should be actionable for the correct group of users. Users must be able to react to information at the appropriate level e.g. tactical or strategical level.

25/26
Week4-Threat Modelling

Step-3: Threat Identification: Cont'

Task-A: Identify Threat Vectors:

  • Threat vectors are paths through which an attacker can exploit to penetrate a system component or bypass defences

    width:1OO% center


25/26
Week4-Threat Modelling

Step-3: Threat Identification: Cont'

Task-B: List Possible Threat Events:

  • Threat events can be characterised by the tactics techniques and procedures (TTP) employed by the attacker.
    width:1OO% center
  • STRIDE (Today), PASTA, DREAD, OCTAVE, NIST SP 800-154,

25/26
Week4-Threat Modelling

Step-3: Threat Identification: Cont'

width:1OO% center


25/26
Week4-Threat Modelling

STRIDE (TTP)

Model for identifying computer security threats.

width:1OO% center


25/26
Week4-Threat Modelling
STRIDE-LM Threat Property Definition
S Spoofing Authentication Impersonating someone or something
T Tampering Integrity / Access Controls Modifying data or code
R Repudiation Non-repudiation Claiming to have not performed a specific action
I Information Disclosure Confidentiality Exposing information or data to unauthorised individuals or roles
D Denial of Service Availability Deny or degrade service
E Elevation of Privilege Authorisation / Least Privilege Gain capabilities without proper authorisation
LM Lateral Movement Segmentation / Least Privilege Expand influence post-compromise; often dependent on Elevation of Privilege

25/26
Week4-Threat Modelling

Activity - Match Incidents to STRIDE-LM Threat Types (5 mins)

Read each incident below and match it to the correct STRIDE-LM threat type
(S, T, R, I, D, E, or LM) and the relevant security property it violates.
Then briefly explain why you chose it.

Incident Match (Threat Type & Property) Justification
1 Attackers send a fake Microsoft 365 login page to collect university credentials.
2 Malicious code is inserted into a trusted software update before release (supply-chain attack).
3 An employee deletes access logs to hide unauthorised activity.
4 A cloud storage bucket is left public, exposing patient medical data.
5 An e-commerce site becomes unreachable during a massive DDoS attack.
6 A staff member exploits misconfigured permissions to gain admin access.
7 Ransomware spreads from one department’s server to another across the network.
8 A hacker changes payroll account numbers to redirect payments.
9 A malicious insider shares confidential tender documents with a competitor.
10 Attackers install a hidden remote access tool that allows them to move between servers.

25/26
Week4-Threat Modelling

Step-4 Attack Modelling:

Mapping sequence of attack, describing tactics, techniques and procedures

  • Describes attacker’s intrusion approach so that users can identify mitigation controls needed to defend the system and prioritise its implementation.

  • To model the attack, you can use either:

    • MITRE ATT&CK:comprehensive matrix of tactics and techniques used by attackers to compromise an organisation’s system

    OR

    • or Cyber Kill Chain: series of well-defined sequence of stages that the attackers are likely to complete in order to achieve their end objective

25/26
Week4-Threat Modelling

Threats Modeling Task Questions (YOUR JOB)

  • What are we working on?
    • diagrams, assests, trust levels, etc.
  • What can go wrong?
    • STRIDE
  • What are we going to do about it?
    • Countermeasures and Mitigation
  • Did we do a good enough job?
    • Evaluation

25/26
Week4-Threat Modelling

Example: Threat Modeling for GenAI integration.

width:1OO% center

width:1OO% center


25/26
Week4-Threat Modelling

Step-1: Scope Definition

  • Define what part of the system will be threat modeled, focusing on components like input handling, output generation, data storage, and user interaction.

  • Key Questions:

    • What are the primary functions of the system?
    • Who are the system users (both legitimate and potential attackers)?
    • What data does the LLM process, and what are its sources and destinations?

25/26
Week4-Threat Modelling

Step 2: System Decomposition

  • External prompt sources (e.g., users, website content, email bodies, etc.)
  • The LLM model itself (e.g., GPT4,5,O, LLaMA, Claude, etc)
  • Server-side functions (e.g., LangChain tools and components)
  • Private data sources (e.g., internal documents or vector databases)

width:1OO% center


25/26
Week4-Threat Modelling

Step 2: System Decomposition & TB

Trust boundary: the point within a system where different levels of trust or different security policies are enforced/applied.
width:1OO% center

  • TB1: between the external endpoints and the LLM
  • Injection, Cross site scripting
  • Both user's and LLM inputs are considered untrusted
  • Two-way trust boundary

25/26
Week4-Threat Modelling

Step 2: System Decomposition & TB - Cont'

width:1OO% center

  • TB2: considered when building applications around LLMs
  • TB3: LLMs themselves do not adhere to authorisation controls internally

25/26
Week4-Threat Modelling

Step 3 : Threat Identification: STRIDE

width:1OO% center

For each TB we are going to have:

  • Strengths and weaknesses for each STRIDE item and for each compents
    width:1OO% center

  • List of vulnerabilities
    width:1OO% center


25/26
Week4-Threat Modelling

Step 3 - Threat Identification: STRIDE - TB1

width:1OO% center

  • For the External points
Category Strengths Weaknesses
1-Spoofing V1: Modify System prompt (prompt injection)
2-Tampering V2: Modify LLM parameters (temperature, length, model, etc.)
3-Repudiation Proper authentication and authorisation (assumed)
4-Information Disclosure V3: Input sensitive information to a third-party site (user behavior)
5-Denial of Service
6-Elevation of Privilege Proper authentication and authorisation (assumed)

25/26
Week4-Threat Modelling

Threat Identification should be carried out for each TB


25/26
Week4-Threat Modelling

1. Can we consider hallucinations as a vulnerability?

Use the following to discuss

2. What about training data poisoning, bias, or hate speech?


25/26
Week4-Threat Modelling

Recommendations

REC_ID Recommendations for Mitigation
REC1 Avoid training LLMs on non-public data. Treat all LLM output as untrusted and enforce data/action restrictions based on LLM requests.
REC2 Limit exposed API surfaces to external prompts. Treat all external inputs as untrusted, applying filtering where needed.
REC3 Educate users on safe behavior during signup and provide consistent notifications when connecting to the LLM.
REC4 Do not train LLMs on sensitive data. Enforce authorisation controls at the data source level, not within the LLM.
REC5 Treat LLM output as untrusted; apply restrictions before using it in other functions to mitigate malicious prompt impact.
REC6 Filter server-side function outputs and sanitise sensitive information before retraining or returning output to users.
REC7 Treat LLM access like typical user access. Enforce authentication/authorisation controls prior to data access, as LLMs cannot protect sensitive information.

25/26
Week4-Threat Modelling

Step 4 - Attack Modelling: to assess the work

  • To model the attack, you can use either:

    • MITRE ATT&CK:comprehensive matrix of tactics and techniques used by attackers to compromise an organisation’s system

    OR

    • or Lockheed Martin Kill Chain: series of well-defined sequence of stages that the attackers are likely to complete in order to achieve their end objective.
      • Steps below

25/26
Week4-Threat Modelling

Threat Modeling vs Attack Modeling

Aspect Threat Modeling (STRIDE) Attack Modeling (MITRE ATT&CK / Kill Chain)
Purpose Identify potential threats in system design Describe actual attacker behaviours and campaigns
Perspective Defender / architect - “What could go wrong?” Adversary / analyst - “How do they attack?”
Stage Design-time → prevention & hardening Run-time / post-incident → detection & response
Focus Violated security properties (Spoofing, Tampering, etc.) Tactics & techniques (Recon, Exploitation, C2 …)
Outcome Threat list + mitigations Attack chain mapping + detection / response gaps

In a nutshell:

STRIDE helps you build securely,
MITRE ATT&CK / Kill Chain help you detect and respond effectively.


25/26
Week4-Threat Modelling

Others

Lab

  • Threat Modelling Lab, see here.

25/26

## Threat Modelling Methodologies: **Asset-Centric Approach** - `Protect` valuable `assets`. - `Focuses` on identifying and `safeguarding` critical `assets` such as data, intellectual property, user information, and physical components. 1. **Identify Assets:** List all valuable assets needing protection. 2. **Assess Impact:** Determine the impact of potential threats to these assets. 3. **Determine Controls:** Implement security measures to protect these assets. - **Benefits:** Ensures critical `components` are `adequately protected`. - **Challenges:** May **overlook** **vulnerabilities** that do **not directly** threaten the identified assets. --- ## Threat Modelling Methodologies: **Attacker-Centric Approach** - Identify potential `attackers` and their `goals`. - Focuses on understanding the attackers' `motivations`, `capabilities`, and `objectives` to better anticipate and defend against their actions. 1. **Profile Attackers:** `Define` potential `attackers` based on motivation, skill `level`, and `resources`. 2. **Understand Goals:** Identify what `attackers` `aim` to achieve (e.g., financial gain, disruption, data theft). 3. **Simulate Attacks:** Model `how` these attackers might `exploit` vulnerabilities. - **Benefits:** Prepares defenses `tailored` to specific `threats`. - **Challenges:** Requires ongoing updates to address evolving threats and techniques. --- ## Threat Modelling Methodologies: **System-Centric Approach** - `Focus` on `vulnerabilities` within the system’s architecture. - `Examines` the system’s `design` and interactions to `identify` and mitigate potential `weaknesses`. 1. **Analyse Architecture:** Map out the system `architecture` and `components`. 2. **Identify Vulnerabilities:** Look for `weaknesses` within the `design` and `implementation`. 3. **Mitigate Risks:** Develop and apply `controls` to `address` `vulnerabilities`. - **Benefits:** Provides a `thorough` `examination` of system structure, helping to identify deep-seated vulnerabilities. - **Challenges:** Can be `complex` and `time-consuming`, especially for large or intricate systems. ---

Users should establish the technical scope, system architecture, and system components before performing threat modelling for a system.

- developers and end-users in various industries

- Anyone

- LLMs process diverse data including user inputs and large text corpora, with data flowing from user interactions to outputs in applications and databases

### Step 3 - Threat Identification: **STRIDE - TB1** ![width:1OO% height:200px center](../../figures/TB_VUL.png) - **For LLMs** | **Category** | **Strengths** | **Weaknesses** | |----------------------------|-----------------------------------------|-------------------------------------------------------------| | **1-Spoofing** | - | -| | **2-Tampering** | - | - | | **3-Repudiation** | -| - | | **4-Information Disclosure** |- | **V4**: LLMs are unable to filter sensitive information (open research) | | **5-Denial of Service** |-|-| | **6-Elevation of Privilege** | - | - | --- ### Step 3 - Threat Identification: **STRIDE - TB1** #### List of vulnerabilities | V_ID | Description | E.g., | |--------|---------------------------------------------------|-----------------------------------------------------------------------------------------------------------| | V1 | Modify System prompt (prompt injection) | Users can modify the system-level prompt restrictions to "jailbreak" the LLM and overwrite previous controls in place | | V2 | Modify LLM parameters (temperature, length, model, etc.) | Users can modify API parameters as input to the LLM such as temperature, number of tokens returned, and model being used. | | V3 | Input sensitive information to a third-party site (user behavior) | Users may knowingly or unknowingly submit private information such as HIPAA details or trade secrets into LLMs. | | V4 | LLMs are unable to filter sensitive information (open research area) | LLMs are not able to hide sensitive information. Anything presented to an LLM can be retrieved by a user. This is an open area of research. | --- ### Step 3 - Threat Identification: **STRIDE - TB2** ![width:1OO% height:200px center](../../figures/TB_VUL_2.png) - **LLMs** | **Category** | **Strengths** | **Weaknesses** | |----------------------------|-----------------------------------------|-------------------------------------------------------------| | **1-Spoofing** | - |V5: Output controlled by prompt input (unfiltered)| | **2-Tampering** | - |Output controlled by prompt input (unfiltered)| | **3-Repudiation** | - | - | | **4-Information Disclosure** | - | - | | **5-Denial of Service** | - | - | | **6-Elevation of Privilege** | - | - | --- ### Step 3 - Threat Identification: **STRIDE - TB2** ![width:1OO% height:200px center](../../figures/TB_VUL_2.png) - **For Server-Side Functions** | **Category** | **Strengths** | **Weaknesses** | |----------------------------|-----------------------------------------|-------------------------------------------------------------| | **1-Spoofing** |Server-side functions maintain separate access to LLM from users| - | | **2-Tampering** | - |V6: Server-side output can be fed directly back into LLM (requires filter)| | **3-Repudiation** | - | - | | **4-Information Disclosure** | - |V6: Server-side output can be fed directly back into LLM (requires filter)| | **5-Denial of Service** | - | - | | **6-Elevation of Privilege** | - | - | --- ### Step 3 - Threat Identification: **STRIDE - TB2** #### List of vulnerabilities ![width:1OO% height:100px center](../../figures/TB_VUL_2.png) | V_ID | Description | E.g., | |---------|-----------------------------------------------------------|----------------------------------------------------------------------------------------------------------| | V5 | Output controlled by prompt input (unfiltered) | LLM output can be controlled by users and external entities. Unfiltered acceptance of LLM output could lead to unintended code execution. | | V6 | Server-side output can be fed directly back into LLM (requires filter) | Unrestricted input to server-side functions can result in sensitive information disclosure or server-side request forgery (SSRF). Server-side controls would mitigate this impact. | --- ### Step 3 - Threat Identification: **STRIDE - TB3** ![width:1OO% height:200px center](../../figures/TB_VUL3.png) - **For the LLMs** | **Category** | **Strengths** | **Weaknesses** | |----------------------------|-----------------------------------------|-------------------------------------------------------------| | **1-Spoofing** | - | V5: Output controlled by prompt input (unfiltered) | | **2-Tampering** | - | V5: Output controlled by prompt input (unfiltered) | | **3-Repudiation** | - | - | | **4-Information Disclosure** | - | - | | **5-Denial of Service** | - | - | | **6-Elevation of Privilege** | - | - | --- ### Step 3 - Threat Identification: **STRIDE - TB3** ![width:1OO% height:200px center](../../figures/TB_VUL3.png) - **Private Data Sources** | **Category** | **Strengths** | **Weaknesses** | |----------------------------|-----------------------------------------|-------------------------------------------------------------| | **1-Spoofing** | - | - | | **2-Tampering** | - | - | | **3-Repudiation** | - | - | | **4-Information Disclosure** | - | V7: Access to sensitive information | | **5-Denial of Service** | - | - | | **6-Elevation of Privilege** | - | - | --- ### Step 3 - Threat Identification: **STRIDE - TB3** #### List of vulnerabilities ![width:1OO% height:200px center](../../figures/TB_VUL3.png) | **V_ID** | **Description** | **E.g.,** | |-------------|--------------------------------------------------|--------------------------------------------------------------------------------------------------| | **V5** | Output controlled by prompt input (unfiltered) | LLM output can be controlled by users and external entities. Unfiltered acceptance of LLM output could lead to unintended code execution. | | **V7** | Access to sensitive information | LLMs have no concept of authorisation or confidentiality. Unrestricted access to private data stores would allow users to retrieve sensitive information. | ---

- More resources are availbale on the book section on the Moodle page.