Large financial institution - LLM abuse & prompt injection testing
Anonymized case study: security readiness assessment for a regulated LLM assistant in a financial services environment.
Context
A large financial institution deployed an LLM-powered assistant for customer support and internal operations. The system operates in a regulated environment with strict compliance requirements.
- •Regulated environment with an LLM assistant
- •Customer-facing use cases
- •Integration with financial data systems
- •Strict compliance and security requirements
Focus areas
The assessment focused on identifying failure modes related to prompt injection and unsafe tool use, which pose significant risks in financial services contexts.
- •Prompt injection vulnerabilities
- •Unsafe tool-use failure modes
- •Data leakage risks
- •Compliance gaps in AI governance
Findings
We observed high/medium severity issues where the assistant could be steered into policy and legal non-compliance, produced AML-sensitive guidance, and claimed or attempted actions beyond its authorized capabilities.
What we did
We conducted a black-box LLM security assessment starting with model and integration reconnaissance (to understand whether the assistant relied on a third-party LLM or an in-house setup). We then executed a structured test plan covering prompt injection, boundary bypass, data-exposure paths, and tool/action misuse. Starting with the OWASP Top 10 for LLM Applications as our baseline for automated and 'known pattern' checks, we escalated to manual, iterative adversarial testing to validate real-world abuse scenarios and document evidence-based findings.
Outputs
The assessment delivered actionable findings and evidence-ready documentation for security and compliance teams.
- •Findings summary — a structured overview of observed high/medium risk behaviors (policy/legal, AML-sensitive scenarios, and action boundaries).
- •Evidence log (repro notes) — documented prompts/steps and observed outputs for internal validation.
- •Risk categorization — issues grouped by severity and impact areas (compliance safety, AML misuse, unauthorized/impossible actions).
- •Readout walkthrough — a short session showing how the behaviors were discovered and what they imply for security and risk.