Benchmarks¶
The tables below are the latest recorded snapshot of the default system
(no experiment matrices). They were produced with an xplan harness that is now
kept as optional, gitignored local tooling. Per-run detail:
build/benchmarks/<plan>/<stamp>/ (REPORT.md, ablation.md, metrics.csv).
| Local harness | tests/benchmarks/xplan/ (optional; not distributed) |
| Artifacts | build/benchmarks/<plan>/<stamp>/ (gitignored) |
| Micro baselines (code) | tests/benchmarks/baseline/*.json |
Corpora: quality plans use BEIR scifact (dataset=scifact, 2000 docs × 50
queries) by default. Cache: ~/.cache/yams/benchmarks/<name> (auto-download).
dataset=synthetic is only for ingest/load/ops throughput — not ranking.
Topology construction defaults (when topology rebuild runs):
min_edge_score=0.25, max_component_docs=64, engine connected.
Latest snapshot¶
Stamp: kpi-20260709T031007Z (search quality) · kpi-20260709T003228Z (ingest / load / repair / ops)\
Host: local macOS
Builds: build/release (ingest, repair, quality); build/prepush-macos (load, ops — Catch2)
Ingest — ingest_pipeline (80 docs × 1 KB, synthetic throughput)¶
| Arm | docs/s | wall_ms | complete |
|---|---|---|---|
| baseline | 96.6 | 828 | yes |
| no_kg | 95.6 | 837 | yes |
| no_vectors | 87.0 | 920 | yes |
| no_vectors_no_kg | 97.2 | 823 | yes |
| no_gliner | 97.8 | 818 | yes |
| minimal | 111.0 | 721 | yes |
Artifacts: build/benchmarks/ingest_pipeline/kpi-20260709T003228Z-ingest/
Retrieval load — retrieval_load¶
| Arm | p50_ms | p95_ms | QPS | searches |
|---|---|---|---|---|
| baseline_hybrid | 28.0 | 41.3 | 123 | 189 |
| keyword_only | 1.20 | 5.45 | 362 | 189 |
Artifacts: build/benchmarks/retrieval_load/kpi-20260709T003228Z-load/
Search quality — BEIR scifact (default hybrid)¶
2000 docs · 50 queries · topk=10 · topology routing off (historical control; the current
product default is hybrid_assist + shadow)
| Metric | Value |
|---|---|
| MRR | 0.613 |
| nDCG | 0.634 |
| MAP | 0.608 |
| recall@10 | 0.713 |
| precision@10 | 0.076 |
Artifacts: build/benchmarks/topology_optimize_v2/kpi-20260709T031007Z-topo-optv2b/ (topo_off arm)
Repair — repair_ability (20 faults)¶
| Arm | injected | repaired | status |
|---|---|---|---|
| fts5 | 20 | 20 | ok |
| graph | 20 | 225* | ok |
| embed | 20 | 0 | fail (timeout ~120s) |
| embed_all_ops | 20 | 0 | fail (timeout ~120s) |
* graph arm counts operations, not 1:1 documents.
Artifacts: build/benchmarks/repair_ability/kpi-20260709T003228Z-repair/
Ops / idle — ops_timeline¶
| Arm | ingest docs/s | idle_fraction |
|---|---|---|
| baseline_no_vectors | 212 | 1.0 |
| with_vectors | 212 | 1.0 |
Artifacts: build/benchmarks/ops_timeline/kpi-20260709T003228Z-ops/
Refresh¶
- Prepare the relevant corpus.
- Run equivalent benchmark workloads under a common stamp.
- Replace tables above from each
summary.md/REPORT.md(default arms only). - Update the stamp line.