Benchmark Results

Benchmark Setup

def compute_recalls(pred_ids: np.ndarray, gt_top: np.ndarray, K: int):    nq = gt_top.shape[0]    out = {}    hits = 0    for i in range(nq):        a = pred_ids[i, :K]        b = gt_top[i, :K]        hits += len(set(a.tolist()) & set(b.tolist()))    out[f"recall@{K}"] = hits / (nq * K)    return out

Result 1: Extreme RAM Caps (256MB) are a Hard Boundary for Many DBs

This is the core motivation for brinicle. In a constrained container (MNIST, 256MB RAM, 1 CPU), the following happened. All failures were verified as OOMKilled by Docker.

Result 2: Latency and Memory Profiles under Constrained DB Deployments

Below are snapshots from the constrained HTTP service benchmark runs.

Result 3: Recall versus Latency Tradeoff (ef_search sweep)

Higher recall usually costs more latency. To make that tradeoff explicit, we ran a sweep:

• Dataset: SIFT (1M, 128 dim)
• Resources: 4GB RAM, 2 CPU
• Distance: L2
• Fixed: M=16, ef_construction=200
• Sweep: ef_search [16, 32, 64, 128, 256]

P95 Latency vs Recall

P99 Latency vs Recall

Memory Usage Comparison

Result 4: In-Process Libraries (FAISS, hnswlib, brinicle)

How does brinicle compare when used the same way you would use FAISS or hnswlib, inside one process, with no network overhead.

What to Take Away from These Results

1. Survivability under hard RAM caps matters. In the 256MB MNIST run, multiple database containers were OOMKilled. brinicle completed build and search.
2. Tail latency is a primary metric for search systems. Average latency is useful, but p95 and p99 are where disk-first and constrained environments tend to show problems. That is why the tables and plots emphasize percentiles.
3. The recall-latency curve is the most informative comparison. The ef_search sweep shows how each system behaves as you push toward higher recall.
4. brinicle is positioned as an engine. If you want a full vector database, you should use one. If you want the index layer with a small memory footprint, brinicle is designed for that.

Reproducing the Benchmarks