Detecting leaked code and brand abuse at scale
A staged verification pipeline: cheap filters first, expensive models last, and every verdict traceable to evidence.
Public code-hosting platforms see a continuous stream of new repositories and commits. A small fraction of them expose proprietary source code, credentials, infrastructure details or brand assets, or impersonate a company. This post is the architecture I use when I design a system that finds those cases automatically and sends a verified alert to the affected organisation.
The problem
Manual triage cannot keep up. The input is high-volume and almost entirely irrelevant, and analysts who review noisy alerts stop trusting them. The system has to be selective before it is smart.
Requirements
Functional
- Continuously discover new public repositories and repository activity.
- Decide whether an item leaks code, credentials or infrastructure information, or impersonates a brand.
- Produce an alert that carries its evidence, and never alert twice on the same finding.
Non-functional
- High precision. False positives are what destroy trust in an alerting system.
- Elastic. Input is bursty; capacity must follow it.
- Bounded cost. GPU inference cannot run on every item that arrives.
- Auditable. Every verdict must trace back to the evidence behind it.
- Independently scalable stages, so a slow model never stalls discovery.
The core idea: shrink the search space before spending compute
Treat the pipeline as a funnel. Each stage answers a more expensive question than the one before it, and only items that survive reach the next stage.
| Stage | Question it answers | Cost per item |
|---|---|---|
| Keyword prefilter | Could this possibly be relevant? | Very low |
| Context assembly | What evidence do we have for it? | Medium (fetch, embed, screenshot) |
| Verification models | Does the evidence show a leak or abuse? | High (model inference) |
| Judge | Do the independent verdicts agree? | Low (reads structured output only) |
Architecture
Component walk-through
1. Discovery and prefilter
Monitoring agents poll and stream public activity and emit raw events. They hold no business logic, so they can be restarted or replaced freely. The keyword prefilter then discards most of the stream before anything expensive runs. It is deliberately crude: its only job is to be cheap and to protect everything downstream.
2. Multimodal context assembly
Text alone cannot reliably say whether a repository impersonates a brand. A README can claim anything; a screenshot of the rendered project shows what a visitor would actually see. So each candidate gets an evidence bundle with two parts:
- Code evidence: files are embedded and indexed in a vector database, so the code model can retrieve the semantically relevant snippets instead of reading a whole repository.
- Visual evidence: a headless browser captures screenshots of the README and any rendered frontend, and stores them in object storage. The queue carries references, not the images.
3. Buffering and dispatch
Evidence bundles are published to partitioned Kafka topics. Keying by repository keeps events for one repository ordered, the log absorbs traffic spikes, and ingestion is decoupled from inference. Redis job queues then hand work to worker pods with low latency. Kafka is the durable buffer; Redis is the fast dispatcher.
4. Three models, three jobs
Instead of one large model doing everything, each model has one narrow job and produces structured output:
| Model | Reads | Looks for |
|---|---|---|
| Vision (7B) | Screenshots from object storage | Logo misuse, cloned UIs, tokens visible in images |
| Code (24B) | Retrieved snippets from the vector index | Hardcoded secrets, proprietary endpoints, internal logic |
| Judge (7B) | The two structured verdicts | Agreement, conflict, and a final confidence score |
The judge never sees raw evidence. It adjudicates between two independent opinions, which keeps it cheap and makes its decisions easy to audit.
5. State and scaling
PostgreSQL stores incident state, repository and commit identifiers, classification output and the audit trail. A (repository, commit) key lets the system check what it has already reported before it alerts. Worker pods run on Kubernetes with a Horizontal Pod Autoscaler driven by queue depth and Kafka lag, not CPU. Backlog is the signal that matters, because CPU stays low while a queue grows.
Failure modes
This table is my reasoning about the design, not an incident log.
| Failure | Effect | Mitigation |
|---|---|---|
| Burst of new repositories | Queues grow | Kafka absorbs it; autoscaler adds workers on lag; the prefilter bounds inflow. |
| Worker dies mid-job | Job could be lost | Keep the job queued until acknowledged and make writes idempotent on (repository, commit). |
| Same repository seen repeatedly | Duplicate alerts | State table is checked before any alert is sent. |
| Vision and code models disagree | Ambiguous verdict | The judge weighs both; low confidence stays under the alert threshold. |
| Model endpoint slows down | Latency rises | Backpressure shows up as queue depth, which the autoscaler reacts to, instead of upstream timeouts. |
Trade-offs
- Precision over recall. The prefilter can miss things, and the reported number measures precision only. I make no recall claim.
- Latency for accuracy. Three model calls plus a judge take longer than one call. The stages are asynchronous, so the cost is delay, not throughput.
- Cheap filter, brittle filter. Keyword matching is easy to evade; its value is cost control, not security.
Result
In production this design reached over 95% precision on newly leaked repositories. That figure is my team's internal measurement, not a published benchmark.
Technology stack
| Layer | Technology |
|---|---|
| Discovery and filtering | Monitoring agents, keyword preprocessing |
| Semantic retrieval | Dense code embeddings, Qdrant |
| Visual evidence | Headless browser, S3-compatible object storage |
| Streaming and queuing | Apache Kafka, Redis |
| Verification | Qwen 7B (vision), Mistral 24B (code), Mistral 7B (judge) |
| State and audit | PostgreSQL |
| Orchestration | Kubernetes with HPA |