Vuln Bank Part 5: Measuring What the Bank Remembers

01_THE_HONEST_STARTING_POINT

At the end of Part 3, I had a number: recall@5 over twenty hand-written queries and twenty known records. That was a useful moment. It proved the real pipeline could retrieve something other than the wording it had just ingested.

It did not prove the bank was good.

Twenty queries are a smoke test. They do not represent the vocabulary, ambiguity, filters, misspellings, product names, payload fragments, or half-formed observations bit will produce during a real hunt. The harness runs when I remember to run it, prints one aggregate, and leaves no history. If recall drops next week, I cannot tell whether the cause was an embedding model, a reranker, a candidate limit, a new namespace, or a change in the corpus.

Before comparing models, I need observable and reproducible evaluations that define what an improvement looks like.

02_RECALL_AS_A_TIME_SERIES

The current harness has one gold record g_i for each query q_i. Its reported recall@k is therefore numerically equal to success@k. Each query contributes 1 when its gold record appears in the top k, otherwise it contributes 0:

1success@k = (1 / N) × Σ 𝟙[g_i ∈ Retrieved_k(q_i)], for i = 1..N2 3where:4N                         = number of evaluation queries5Retrieved_k(q_i)          = first k records returned for query q_i6𝟙[condition]              = 1 when condition is true, otherwise 0

For the current twenty-query set, each miss changes success@k by 1/20 = 0.05, or five percentage points. That is why I should not read a small score change as a precise estimate of production quality.

That stays as the headline because it maps directly to the product: bit receives five records, so an expected methodology outside the top five is unavailable to it. But I want every evaluation run to record more than one number:

• success@1, @3, @5, and @10 to show whether a change improves retrieval or merely pushes the same answer around;
• MRR@10 to make rank movement visible when the gold record is present;
• candidate recall before reranking to separate retrieval failures from reranker failures;
• dense-only, lexical-only, fused, and reranked results for a small ablation on every run;
• p50 and p95 latency for embedding, both database legs, fusion, reranking, and end to end;
• candidate counts and eligible-row counts so a short dense list is not automatically blamed on HNSW;
• model revision, serving image, dimensions, corpus version, evaluation-set version, and all retrieval settings so two runs are actually comparable.

The unit I care about is an evaluation run, not a log line. Each run should have an id and immutable configuration, with per-query results underneath it:

1{2  "run_id": "2026-08-19T18:42:11Z",3  "corpus_version": "seed-0002",4  "eval_set": "security-retrieval-v2",5  "embedding": "BAAI/bge-m3@<revision>",6  "reranker": "BAAI/bge-reranker-v2-m3@<revision>",7  "candidate_k": 30,8  "fused_k": 20,9  "output_k": 5,10  "hnsw_ef_search": 100,11  "success_at_5": 0.0,12  "p95_ms": 013}

Those zeroes are deliberate placeholders. I do not want to publish a benchmark result I have not reproduced and stored.

Once runs are persisted, the useful view is a graph of recall and latency over time, annotated with model, corpus, and configuration changes. A regression threshold can then fail CI or block a deployment. At first I will use both an absolute floor and a “no worse than baseline” check, while accounting for the coarse five-point steps in this small evaluation set.

03_BUILDING_A_TEST_SET_I_CAN_TRUST

Calculating the metric is straightforward. Building a representative gold set is harder.

I want to grow it in layers:

• Curated paraphrases: queries that describe an observation without repeating the record title or vulnerability class.
• Lexical needles: CVE ids, header names, parameters, framework symbols, payload fragments, and punctuation-heavy tokens that English FTS may split badly.
• Confusable negatives: techniques that share vocabulary but require different preconditions, such as cache poisoning versus host-header password-reset poisoning.
• Filtered cases: the same query under different namespace, CWE, product, and quality-tier filters.
• No-answer cases: queries for which the bank has no suitable methodology, so returning a plausible neighbor is not counted as success.
• Queries from real use: sanitised and reviewed traces of what bit actually asked, especially misses and reformulations.

One gold id per query is convenient but often false. A methodology question can have several useful answers, and relevance can be graded. I eventually want judgments such as essential, useful, related, and irrelevant, made without looking at which model produced the candidate. That opens the door to nDCG and gives the reranker a much fairer test.

I also need a frozen holdout. If every miss becomes a new test and I tune against all of them, I will build an excellent retriever for yesterday’s mistakes. A development set is for choosing knobs; the holdout is for checking whether those choices generalise.

04_THE_KNOBS_BEFORE_THE_MODELS

Model swaps are expensive and visible, so they attract attention. Several cheaper knobs may matter just as much:

I will change one family at a time. First candidate widths and ef_search, then representation fields, then lexical analysis, then models. Every run gets the same corpus and gold set, a warm-up, repeated latency samples, and a cold-start measurement. A configuration only wins if the gain survives across vulnerability classes and still fits the operational budget.

For the dense representation, I want to test adding title, when_to_use, and confirmation_signal instead of embedding only symptom + procedure. For lexical retrieval, I want to separate natural-language FTS from literal identifiers, perhaps with a simple dictionary or trigram index. Both experiments are cheaper than replacing the model and may improve retrieval on their own.

05_WHY_BGE_M3_IS_THE_BASELINE

The current pair is BAAI/bge-m3 for 1024-dimensional dense embeddings and BAAI/bge-reranker-v2-m3 for cross-encoder reranking.

I did not choose them because I had proven they were the best cybersecurity models. I chose them because they were a credible, compatible pair that I could serve locally on CPU through Hugging Face Text Embeddings Inference. BGE-M3 supports long inputs and many languages, its 1024 dimensions fit pgvector cleanly, and the reranker is described by its authors as the lightweight member of their v2 family.

They also ran on the VPS I already had, which was a firm requirement.

Even then, the default TEI token budget caused an out-of-memory kill during model loading. Both services now use a 2048-token batch cap and automatic truncation. Any alternative has to fit the same deployment budget; a small recall improvement would not justify requiring a GPU or adding several seconds to every query.

I also needed to make a decision and keep moving. It is easy to spend two weeks reading embedding leaderboards before the system has one trustworthy evaluation query. BGE-M3 gave me a practical baseline that I can now compare against.

One correction to my earlier language: only the model IDs and dimensions are fixed today. The production compose files still use cpu-latest, and no Hugging Face commit revision is pinned. Pinning both is part of the observability work because a baseline that can change underneath me is not a baseline.

06_MODELS_WORTH_TESTING

There are two different ideas hiding behind “a cybersecurity model,” and I need to keep them separate.

Retrieval models: jinaai/jina-embeddings-v3 provides task-specific retrieval adapters and configurable dimensions. NovaSearch/stella_en_1.5B_v5 is an English retrieval model with Matryoshka dimensions. Both are plausible candidates without additional training. Stella’s roughly 1.5B-class backbone is much heavier than BGE-M3, so CPU memory and latency may rule it out. Jina v3 also changes the query/document prompting contract, which the experiment must apply correctly.

Cybersecurity encoders: ehsanaghaei/SecureBERT and its newer Cisco successor were pretrained on cybersecurity text. Vocabulary around malware families, indicators, and threat intelligence should be more familiar to them than to a general encoder. The published SecureBERT checkpoint is a masked-language-model backbone rather than a sentence-retrieval model, however, so mean-pooling its hidden states would not provide a fair comparison.

The interesting experiment is to use a cybersecurity encoder as the starting point for contrastive fine-tuning on (query, relevant methodology, hard negative) triples. That could teach both the domain language and the retrieval objective. It also creates a model I have to train, version, evaluate, and maintain, so it comes after the benchmark, not before it.

The reranker has a similar ladder:

• keep bge-reranker-v2-m3 as the CPU-friendly baseline;
• test a small English cross-encoder such as cross-encoder/ms-marco-MiniLM-L-6-v2 for latency, accepting that web-search training is not security training;
• test the larger BGE v2 Gemma or MiniCPM rerankers if hardware allows, since BAAI positions them above v2-m3 for quality but at a much higher serving cost;
• fine-tune a DeBERTa or SecureBERT-family sequence classifier on my graded security relevance pairs, with hard negatives from the current retriever.

Domain adaptation may help distinguish two records that both mention “Host header” but have different exploit preconditions, or interpret a terse symptom that never names the vulnerability. It may also overfit to familiar jargon and perform worse on the messy language bit actually uses. A holdout and per-category breakdown should reveal that failure mode.

07_THE_EXPERIMENT_ORDER

To keep this from becoming an endless model bake-off, I want a fixed order:

1. Freeze the baseline. Pin model revisions and inference images; persist current per-query outputs and latency.
2. Grow the evaluation set. Add confusable, filtered, literal-token, multilingual, and no-answer cases; keep a holdout.
3. Instrument the pipeline. Store stage timings, candidate ids and ranks, pool sizes, filter selectivity, truncation, and configuration.
4. Tune cheap knobs. Candidate widths, ef_search, FTS behavior, and which fields each stage sees.
5. Run ablations. Dense only, lexical only, fusion without reranking, and the full pipeline.
6. Swap general retrieval models. Re-embed into a parallel index, compare quality and resource use, then discard or promote.
7. Try domain adaptation. Fine-tune only after the error set is large enough to tell me what the model needs to learn.

Parallel indexes matter here. Replacing the live vector(1024) column in place would turn an experiment into a migration. A model-versioned embedding table lets old and new models retrieve the same record ids side by side, supports shadow evaluation, and gives me a rollback path.

08_WHAT_SUCCESS_LOOKS_LIKE

I want a retrieval system whose behavior I can explain and whose resource use matches the real deployment:

• I can see when success@5 regresses and which query categories moved.
• I can tell whether the miss happened during candidate generation, fusion, or reranking.
• I know the latency and memory cost of every claimed improvement.
• I can reproduce a run from pinned code, models, data, and configuration.
• I can change models without overwriting the last known-good index.

The next phase is to record misses, understand where they happen, and use that evidence to decide whether a new model is worth operating.