Open-vocab document & figure classification · benchmark
Given a document or figure image and an arbitrary set of text labels, which model picks the right one? Dual-encoder (CLIP-family) models are ranked by top-1 accuracy over a ~48-label open vocabulary on the open doc-openvocab-benchmark. This is the open-vocabulary task a fixed-label classifier structurally can't do — the labels are supplied at inference, not baked into a head.
Retention = Synonym ÷ Broad, on the same images and the same 48-way choice — how much accuracy survives when the labels are reworded. Read it as a robustness dial: ≈1.0 means the model matched the concept and is indifferent to exact wording; well below 1.0 means it leaned on the specific label strings and loses accuracy under paraphrase. It can edge slightly above 1.0 when a model happens to prefer the paraphrase wording. (Because both terms use the identical 48-label set, the ratio is a fair comparison — not inflated by candidate-set size.)
Highlighted row is the Nutrient model. open = downloadable weights; commercial = contact us; non-commercial = competitor weights under a research-only license. General CLIP/SigLIP models are the baselines; ours is the only document-adapted one. Params = approximate total (image + text towers). Speed = images/s on the standard hardware. Every model uses the same prompt policy; top-1 is argmax over per-label logits (rank-invariant to SigLIP sigmoid vs CLIP softmax).
Scoring is self-service and reproducible — no server in the loop. Score any dual-encoder (CLIP/SigLIP-family) model and open a PR with the results JSON.
pip install torch transformers open_clip_torch timm datasets python -m benchmark.score --model <your-model> --benchmark-repo nutrientdocs/doc-openvocab-benchmark
The reference scorer benchmark/score.py and the self-scoring
benchmark_meta.json (candidate labels + prompt policy) travel with the dataset, so the
numbers here reproduce from public artifacts alone.
Models benchmarked: nutrientdocs/doc-img-classification · apple/DFN5B-CLIP-ViT-H-14-378 · apple/MobileCLIP2-S4 · facebook/metaclip-2-worldwide-huge-quickgelu · google/siglip2-base-patch16-512 · google/siglip2-so400m-patch14-384 · laion/CLIP-ViT-H-14-laion2B-s32B-b79K · openai/clip-vit-large-patch14-336 · timm/eva02_large_patch14_clip_336