Open source
Prompt-injection classifier
An open-source classifier that sorts incoming text into five categories of attack before it reaches an AI model.
The Challenge
Prompt injection is the most common way AI applications get manipulated. Keyword filters miss rephrased attacks, and asking the AI model itself to spot attacks is slow and unreliable.
How It Works
1
Text is classified before it reaches the model.
Each input is labeled clean, instruction override, indirect injection, jailbreak or tool-call hijacking.
2
The model is fine-tuned for the job.
A transformer model is fine-tuned on a balanced dataset, built with a pipeline that uses the model itself to validate new training examples.
3
Results are tracked openly.
Training runs and evaluation results are published, with performance measured for each attack category on held-out data.
4
It runs as a service.
A lightweight API lets other applications check text before passing it on.
Built With
Hugging Face Transformers, PyTorch, GPU training, Weights & Biases, FastAPI
What it's designed to do
- Catch manipulation attempts before they reach an AI model
- Tell different kinds of attack apart, so each can be handled correctly
- Give teams a fast, inspectable guard they can run themselves
Want something like this for your business?
Every build starts with an assessment, so we can adapt this pattern to how you actually work.