Enterprise AI Analysis

Existing benchmarks for structured output either focus on schema compliance (JSONSchemaBench, StructEval, DeepJSONEval) or value correctness within a single domain (ExtractBench, LLMStructBench). Constrained decoding methods improve syntactic validity but can degrade semantic accuracy. General multi-modal benchmarks (MMBench, MMMU) address broad reasoning, while domain-specific ones (DocVQA, ChartQA, AudioBench) focus on QA but don't require schema-compliant JSON output with grounded per-field value accuracy. SOB unifies these dimensions, evaluating multi-source extraction (text, images, audio) with value-level accuracy and cross-source comparison.

SOB processes three open-source datasets: HotpotQA (text), olmOCR-bench (images), and AMI Meeting Corpus (audio). For each record, a human-authored JSON schema and ground-truth structured output are verified by an LLM reviewer. All models receive text-normalized context, ensuring structured-output capability is isolated from raw vision/speech processing quality. Schemas are categorized as medium (depth 2) or hard (depth ≥3), with most being hard. The evaluation pipeline checks for parse validity, schema compliance, and then compares path-flattened leaf nodes against ground truth using seven metrics. A key example illustrates multi-hop reasoning and nested structure extraction.

SOB employs seven metrics: JSON Pass Rate, Value Accuracy (primary, exact leaf-value match), Faithfulness (token F1, partial credit), Path Recall (structural completeness), Structure Coverage (structural precision/recall), Type Safety (JSON type correctness), and Perfect Response (exact full-object match). Semantic metrics are 'hardened' by a factor that gates on structural correctness and a coverage gate. These metrics are grouped into categories like Long Context Extraction, Complex Schema Handling, and Output Contract Reliability. Aggregation uses schema-complexity-weighted means, as complex schemas are more representative of production tasks. Inference is greedy (temperature 0.0), with reasoning disabled where possible to isolate extraction capability.

Evaluation of 21 models reveals consistent patterns: high JSON Pass Rate (near-perfect schema compliance) but significantly lower Value Accuracy, indicating that producing schema-valid JSON is easier than extracting correct grounded values. The gap between these metrics is 15-25 percentage points. Value Accuracy drops sharply across modalities: 83.0% for text, 67.2% for images, and 23.7% for audio. Model rankings shift across modalities, and model size doesn't predict structured output quality. Structured hallucinations are hard to detect because they appear structurally correct. The benchmark shows that schema constraints have a smaller effect on grounded extraction than on structural reliability.

JSON Pass Rate vs. Value Accuracy Across Models

SOB's ground truth is human-authored with LLM cross-checks, reducing LLM-as-generator bias, but schemas reflect team conventions. Future versions will incorporate production-standard schemas. The evaluation isolates structured extraction from raw vision/ASR quality by using text-normalized contexts, meaning audio results are an upper bound (not including ASR error). Metrics are strict (exact-match, ordered arrays), favoring precision. Future work includes semantic-aware comparison, order-sensitive flags, sensitivity analysis, constrained decoding baselines, additional modalities (video, code), and a live leaderboard.

SOB addresses the critical need for evaluating structured output quality in LLMs by focusing on the correctness of values within JSON, not just schema compliance. It highlights a significant gap: models achieve high schema compliance but struggle with value accuracy, especially across diverse modalities like text, images, and audio. The benchmark demonstrates that schema compliance alone is an insufficient measure of structured output quality. By releasing the dataset, evaluation pipeline, and code, SOB aims to enable the community to build and measure what truly matters for production structured output: getting the values right.