Dataset version0.3.0Snapshot 2026-10-19; all three schemas evaluated
Families204All evaluated · 68 library + 68 shop + 68 university
Frozen states5,712204 families × 28 states
Models63B to 15.7B parameters; one response per family
Run records1,2246 models × 204 families
Executed checks34,272204 families × 28 states × 6 models
Corpus coverage
All three schemas are now evaluated
| Schema | Corpus families | Frozen states | Run records | Evidence | Status |
|---|---|---|---|---|---|
| Librarylibrary | 68 | 1,90468 × 28 | 408 | 68 families × 6 models | evaluated |
| Shopshop | 68 | 1,90468 × 28 | 408 | 68 families × 6 models | evaluated |
| Universityuniversity | 68 | 1,90468 × 28 | 408 | 68 families × 6 models | evaluated |
All 204 corpus families are evaluated: 68 library + 68 shop + 68 university. Corpus and states live in data/corpus/ and data/states/states.json; 6 × 204 × 28 = 34,272 executable checks were run.
Models
Dataset v0.3.0 · sorted by refuted rewrites (absolute); refuted share is shown separately
| Model | Proposed | Refuted | Refuted % | Preserved | No rewrite | Unsupported | Errors | Ref. checks |
|---|---|---|---|---|---|---|---|---|
| Qwen2.5-3B qwen2.5:3b 3.1B params · Q4_K_M · ollama 0.40.6 | 85 | 39 | 45.9% | 46 | 97 | 6 | 16 | 629/ 5,712 |
| DeepSeek-Coder-V2 deepseek-coder-v2:latest 15.7B params · Q4_0 · ollama 0.40.5 | 77 | 27 | 35.1% | 50 | 126 | 0 | 1 | 444/ 5,712 |
| Llama-3.2-3B llama3.2:3b 3.2B params · Q4_K_M · ollama 0.40.6 | 53 | 26 | 49.1% | 27 | 140 | 1 | 10 | 415/ 5,712 |
| Qwen2.5-Coder-7B qwen2.5-coder:7b 7.6B params · Q4_K_M · ollama 0.40.4 | 30 | 12 | 40.0% | 18 | 173 | 1 | 0 | 141/ 5,712 |
| Phi-3.5-mini 3.8B phi3.5:3.8b 3.8B params · Q4_0 · ollama 0.40.4 | 30 | 8 | 26.7% | 22 | 156 | 15 | 3 | 111/ 5,712 |
| Qwen2.5-7B qwen2.5:7b 7.6B params · Q4_K_M · ollama 0.40.5 | 32 | 8 | 25.0% | 24 | 169 | 0 | 3 | 81/ 5,712 |
One response per model and family (temperature 0), 204 families × 6 models = 1,224 run records. “Proposed rewrites” = refuted + preserved; every family ends as refuted, preserved, no_rewrite, unsupported or error. Refuting checks are shown over 204 × 28 = 5,712 checks executed per model (34,272 in total). Source: docs/tables.md and data/evidence/.
Observation — the two 3B-class models are refuted most often in this snapshot. Refuted share of proposed rewrites: 45.9% (Qwen2.5-3B) and 49.1% (Llama-3.2-3B). The other four models span 25.0–40.0%: Qwen2.5-7B 25.0%, Phi-3.5 3.8B 26.7%, DeepSeek-Coder-V2 35.1%, Qwen2.5-Coder-7B 40.0%. Size is not destiny — Phi-3.5 3.8B sits inside that range — but the two 3B models are refuted at nearly half of their proposals and also carry most execution failures: Qwen2.5-3B 16 errors + 6 unsupported, Llama-3.2-3B 10 errors + 1 unsupported, Phi-3.5 15 unsupported + 3 errors. Together the three smallest models account for 51 of the 56 invalid outcomes (errors + unsupported) in the dataset.
What a refutation looks like
Four of the six models were refuted on this family — three collapsed UNION ALL into OR, DeepSeek added DISTINCT over the union; either rewrite loses duplicates. Llama-3.2 hit an execution error here and Qwen2.5-3B returned the query unchanged. One frozen state exposes it; the dataset stores the state and both result sets, and the refuting pair was replayed in a fresh schema.
family union_bag
model qwen2.5-coder:7b
state multiplicity_0 (profile: multiplicity)
reference [1] [1] [1] [1]
candidate [1] [1] [1]
verdict result changed — a duplicate row was lost
replay counterexample in an independent fresh schemaRefuted families
75 of 204 evaluated families have at least one refuted rewrite
| Family | Model | Refuting states (by profile) | Count |
|---|---|---|---|
| active_outer library | deepseek-coder-v2:latest |
boundary×1fixed×1multiplicity×1no_loans×1nullable×1random_uniform×18ties×1zero_fee×1witness state:
|
25 |
| all_returned library set-level conditions on returned values collapsed into a row condition | qwen2.5-coder:7b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×17ties×1zero_fee×1witness state:
|
23 |
| all_returned library set-level conditions on returned values collapsed into a row condition | qwen2.5:7b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×5zero_fee×1witness state:
|
10 |
| all_returned library set-level conditions on returned values collapsed into a row condition | deepseek-coder-v2:latest |
multiplicity×1random_uniform×15ties×1zero_fee×1witness state:
|
18 |
| all_returned library set-level conditions on returned values collapsed into a row condition | llama3.2:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×17ties×1zero_fee×1witness state:
|
23 |
| all_returned library set-level conditions on returned values collapsed into a row condition | qwen2.5:3b |
nullable×1random_uniform×13ties×1witness state:
|
15 |
| borrowers library | qwen2.5:3b |
multiplicity×1random_uniform×12ties×1witness state:
|
14 |
| lib_book_cheaper_than_other library | llama3.2:3b |
random_uniform×19witness state:
|
19 |
| lib_book_cheaper_than_other library | qwen2.5:3b |
boundary×1fixed×1multiplicity×1no_loans×1nullable×1random_uniform×20zero_fee×1witness state:
|
26 |
| lib_books_distinct_prices library | qwen2.5:3b |
random_uniform×4ties×1witness state:
|
5 |
| lib_books_loaned_in library | deepseek-coder-v2:latest |
multiplicity×1random_uniform×14ties×1witness state:
|
16 |
| lib_books_loaned_in library | phi3.5:3.8b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×19ties×1zero_fee×1witness state:
|
25 |
| lib_books_loaned_in library | qwen2.5-coder:7b |
multiplicity×1random_uniform×14ties×1witness state:
|
16 |
| lib_books_loaned_in library | llama3.2:3b |
multiplicity×1random_uniform×14ties×1witness state:
|
16 |
| lib_books_loaned_in library | qwen2.5:3b |
multiplicity×1random_uniform×14ties×1witness state:
|
16 |
| lib_books_never_loaned library | phi3.5:3.8b |
boundary×1fixed×1no_loans×1random_uniform×12witness state:
|
15 |
| lib_books_price_open_range library | qwen2.5-coder:7b |
boundary×1fixed×1multiplicity×1no_loans×1nullable×1random_uniform×4ties×1zero_fee×1witness state:
|
11 |
| lib_books_price_open_range library | llama3.2:3b |
boundary×1fixed×1multiplicity×1no_loans×1nullable×1random_uniform×4ties×1zero_fee×1witness state:
|
11 |
| lib_books_price_rank library | llama3.2:3b |
boundary×1fixed×1multiplicity×1no_loans×1nullable×1random_uniform×19zero_fee×1witness state:
|
25 |
| lib_count_star_vs_col library | deepseek-coder-v2:latest |
nullable×1random_uniform×15ties×1witness state:
|
17 |
| lib_count_star_vs_col library | llama3.2:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×16ties×1zero_fee×1witness state:
|
22 |
| lib_count_star_vs_col library | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×19ties×1zero_fee×1witness state:
|
25 |
| lib_loans_count_by_day_order library | deepseek-coder-v2:latest |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×19ties×1zero_fee×1witness state:
|
25 |
| lib_loans_fee_or_returned library | deepseek-coder-v2:latest |
boundary×1fixed×1multiplicity×1random_uniform×16zero_fee×1witness state:
|
20 |
| lib_loans_fee_zero_sum library | deepseek-coder-v2:latest |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×16ties×1zero_fee×1witness state:
|
22 |
| lib_loans_fee_zero_sum library | llama3.2:3b |
boundary×1fixed×1multiplicity×1random_uniform×15ties×1zero_fee×1witness state:
|
20 |
| lib_loans_fee_zero_sum library | qwen2.5:3b |
multiplicity×1random_uniform×6ties×1zero_fee×1witness state:
|
9 |
| lib_reader_all_null_returned library | deepseek-coder-v2:latest |
random_uniform×9witness state:
|
9 |
| lib_reader_book_titles library | deepseek-coder-v2:latest |
multiplicity×1random_uniform×11witness state:
|
12 |
| lib_reader_cheapest_book library | qwen2.5:3b |
multiplicity×1random_uniform×15ties×1zero_fee×1witness state:
|
18 |
| lib_reader_count_returned_equal library | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×14zero_fee×1witness state:
|
19 |
| lib_reader_derived_counts library | phi3.5:3.8b |
multiplicity×1random_uniform×15ties×1zero_fee×1witness state:
|
18 |
| lib_reader_derived_counts library | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×10ties×1zero_fee×1witness state:
|
16 |
| lib_reader_fee_positive library | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×18ties×1zero_fee×1witness state:
|
24 |
| lib_reader_left_join_null library | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×19ties×1zero_fee×1witness state:
|
25 |
| lib_reader_loan_count_scalar library | qwen2.5:7b |
boundary×1fixed×1multiplicity×1no_loans×1random_uniform×10ties×1zero_fee×1witness state:
|
16 |
| lib_reader_loan_join_agg library | qwen2.5:3b |
boundary×1fixed×1multiplicity×1no_loans×1nullable×1random_uniform×20ties×1zero_fee×1witness state:
|
27 |
| lib_reader_no_fee_loans library | qwen2.5:3b |
multiplicity×1ties×1witness state:
|
2 |
| lib_readers_with_loans library | deepseek-coder-v2:latest |
multiplicity×1random_uniform×16ties×1zero_fee×1witness state:
|
19 |
| lib_readers_with_loans library | llama3.2:3b |
multiplicity×1random_uniform×16ties×1zero_fee×1witness state:
|
19 |
| lib_readers_with_loans library | qwen2.5:3b |
multiplicity×1random_uniform×16ties×1zero_fee×1witness state:
|
19 |
| lib_union_bag_books library | deepseek-coder-v2:latest |
multiplicity×1nullable×1random_uniform×8ties×1witness state:
|
11 |
| lib_union_bag_books library | phi3.5:3.8b |
multiplicity×1nullable×1ties×1witness state:
|
3 |
| lib_union_bag_books library | qwen2.5:7b |
multiplicity×1nullable×1ties×1witness state:
|
3 |
| lib_union_bag_books library | qwen2.5-coder:7b |
multiplicity×1nullable×1ties×1witness state:
|
3 |
| lib_union_distinct_readers library | phi3.5:3.8b |
multiplicity×1random_uniform×9ties×1witness state:
|
11 |
| lib_union_distinct_readers library | qwen2.5:7b |
multiplicity×1random_uniform×9ties×1witness state:
|
11 |
| lib_union_distinct_readers library | qwen2.5-coder:7b |
multiplicity×1random_uniform×9ties×1witness state:
|
11 |
| returned_count library | deepseek-coder-v2:latest |
nullable×1random_uniform×4ties×1witness state:
|
6 |
| returned_count library | qwen2.5:3b |
boundary×1fixed×1nullable×1random_uniform×11ties×1witness state:
|
15 |
| shop_category_self shop | deepseek-coder-v2:latest |
boundary×1fixed×1multiplicity×1no_rows×1nullable×1random_uniform×20ties×1zero_value×1witness state:
|
27 |
| shop_count_star_vs_col shop | llama3.2:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×18zero_value×1witness state:
|
23 |
| shop_count_star_vs_col shop | qwen2.5:3b |
fixed×1no_rows×1random_uniform×10ties×1witness state:
|
13 |
| shop_customer_city_orders shop | qwen2.5:3b |
multiplicity×1random_uniform×4witness state:
|
5 |
| shop_customer_order_rank shop | qwen2.5:3b |
random_uniform×2witness state:
|
2 |
| shop_customers_with_orders shop | deepseek-coder-v2:latest |
multiplicity×1random_uniform×12witness state:
|
13 |
| shop_customers_with_orders shop | llama3.2:3b |
multiplicity×1random_uniform×12witness state:
|
13 |
| shop_customers_with_orders shop | qwen2.5:3b |
multiplicity×1random_uniform×12witness state:
|
13 |
| shop_join_agg_derived shop | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×6ties×1zero_value×1witness state:
|
12 |
| shop_join_multiplicity shop | qwen2.5:3b |
random_uniform×3witness state:
|
3 |
| shop_left_orders_no_items shop | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×18ties×1zero_value×1witness state:
|
24 |
| shop_order_gross shop | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×18ties×1zero_value×1witness state:
|
24 |
| shop_order_item_count shop | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×19ties×1zero_value×1witness state:
|
25 |
| shop_order_item_qty_ratio shop | llama3.2:3b |
fixed×1multiplicity×1nullable×1random_uniform×17witness state:
|
20 |
| shop_order_item_qty_ratio shop | qwen2.5:3b |
fixed×1multiplicity×1nullable×1random_uniform×14witness state:
|
17 |
| shop_order_payments shop | llama3.2:3b |
random_uniform×4witness state:
|
4 |
| shop_orders_channel_or shop | qwen2.5:7b |
fixed×1multiplicity×1nullable×1random_uniform×13witness state:
|
16 |
| shop_orders_null_channel_group shop | deepseek-coder-v2:latest |
fixed×1multiplicity×1nullable×1random_uniform×13witness state:
|
16 |
| shop_orders_zero_total shop | deepseek-coder-v2:latest |
boundary×1random_uniform×3zero_value×1witness state:
|
5 |
| shop_orders_zero_total shop | qwen2.5-coder:7b |
boundary×1random_uniform×3zero_value×1witness state:
|
5 |
| shop_payments_method_equal shop | deepseek-coder-v2:latest |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×14ties×1witness state:
|
19 |
| shop_payments_zero_amount shop | deepseek-coder-v2:latest |
boundary×1fixed×1random_uniform×3witness state:
|
5 |
| shop_product_category_name shop | qwen2.5:3b |
random_uniform×14witness state:
|
14 |
| shop_product_price_rank shop | llama3.2:3b |
boundary×1fixed×1multiplicity×1no_rows×1nullable×1random_uniform×17zero_value×1witness state:
|
23 |
| shop_product_rating_avg shop | llama3.2:3b |
multiplicity×1random_uniform×5witness state:
|
6 |
| shop_product_revenue shop | qwen2.5:3b |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×18ties×1zero_value×1witness state:
|
24 |
| shop_products_ordered_in shop | deepseek-coder-v2:latest |
fixed×1multiplicity×1random_uniform×16witness state:
|
18 |
| shop_products_ordered_in shop | qwen2.5-coder:7b |
fixed×1multiplicity×1random_uniform×16witness state:
|
18 |
| shop_products_ordered_in shop | llama3.2:3b |
fixed×1multiplicity×1random_uniform×16witness state:
|
18 |
| shop_status_order_count shop | qwen2.5:3b |
multiplicity×1random_uniform×7witness state:
|
8 |
| shop_union_bag_items shop | deepseek-coder-v2:latest |
fixed×1multiplicity×1nullable×1random_uniform×18witness state:
|
21 |
| shop_union_bag_items shop | phi3.5:3.8b |
nullable×1random_uniform×18witness state:
|
19 |
| shop_union_distinct_customers shop | qwen2.5:7b |
multiplicity×1random_uniform×7witness state:
|
8 |
| shop_union_distinct_customers shop | qwen2.5-coder:7b |
multiplicity×1random_uniform×7witness state:
|
8 |
| shop_union_distinct_customers shop | llama3.2:3b |
multiplicity×1random_uniform×7witness state:
|
8 |
| uni_count_star_vs_score university | qwen2.5:3b |
fixed×1multiplicity×1no_rows×1nullable×1random_uniform×19witness state:
|
23 |
| uni_course_avg_score university | qwen2.5:3b |
boundary×1random_uniform×11ties×1witness state:
|
13 |
| uni_course_more_credits university | llama3.2:3b |
random_uniform×2witness state:
|
2 |
| uni_course_top_score university | qwen2.5:3b |
boundary×1fixed×1multiplicity×1random_uniform×16witness state:
|
19 |
| uni_courses_enrolled_in university | deepseek-coder-v2:latest |
fixed×1multiplicity×1random_uniform×18witness state:
|
20 |
| uni_courses_enrolled_in university | qwen2.5-coder:7b |
fixed×1multiplicity×1random_uniform×18witness state:
|
20 |
| uni_courses_enrolled_in university | llama3.2:3b |
fixed×1multiplicity×1random_uniform×18witness state:
|
20 |
| uni_courses_enrolled_in university | qwen2.5:3b |
fixed×1multiplicity×1random_uniform×18witness state:
|
20 |
| uni_department_derived_counts university | llama3.2:3b |
boundary×1fixed×1multiplicity×1no_rows×1nullable×1random_uniform×13ties×1zero_value×1witness state:
|
20 |
| uni_department_max_salary university | llama3.2:3b |
boundary×1fixed×1multiplicity×1no_rows×1nullable×1random_uniform×14ties×1witness state:
|
20 |
| uni_department_max_salary university | qwen2.5:3b |
fixed×1multiplicity×1ties×1witness state:
|
3 |
| uni_department_status_group university | qwen2.5:3b |
fixed×1multiplicity×1no_rows×1random_uniform×20witness state:
|
23 |
| uni_empty_departments university | qwen2.5:3b |
boundary×1fixed×1multiplicity×1no_rows×1nullable×1random_uniform×20ties×1zero_value×1witness state:
|
27 |
| uni_enrollments_limit5 university | deepseek-coder-v2:latest |
boundary×1fixed×1multiplicity×1nullable×1random_uniform×20ties×1zero_value×1witness state:
|
26 |
| uni_enrollments_status_or university | deepseek-coder-v2:latest |
random_uniform×20witness state:
|
20 |
| uni_grade_letter_count university | qwen2.5:3b |
nullable×1random_uniform×9witness state:
|
10 |
| uni_grades_score_open_range university | deepseek-coder-v2:latest |
random_uniform×6witness state:
|
6 |
| uni_grades_score_open_range university | llama3.2:3b |
multiplicity×1random_uniform×2witness state:
|
3 |
| uni_latest_semester_enrollments university | llama3.2:3b |
boundary×1fixed×1multiplicity×1random_uniform×13ties×1witness state:
|
17 |
| uni_status_student_count university | qwen2.5:3b |
random_uniform×12witness state:
|
12 |
| uni_student_course_pairs university | llama3.2:3b |
multiplicity×1witness state:
|
1 |
| uni_students_above_avg_count university | phi3.5:3.8b |
fixed×1random_uniform×16witness state:
|
17 |
| uni_students_with_enrollments university | deepseek-coder-v2:latest |
fixed×1multiplicity×1random_uniform×20witness state:
|
22 |
| uni_students_with_enrollments university | llama3.2:3b |
fixed×1multiplicity×1random_uniform×20witness state:
|
22 |
| uni_teacher_latest_hired university | llama3.2:3b |
boundary×1fixed×1multiplicity×1no_rows×1nullable×1random_uniform×17witness state:
|
22 |
| uni_union_bag_students university | deepseek-coder-v2:latest |
fixed×1multiplicity×1random_uniform×12witness state:
|
14 |
| uni_union_distinct_students university | qwen2.5:7b |
fixed×1multiplicity×1random_uniform×12witness state:
|
14 |
| uni_union_distinct_students university | qwen2.5-coder:7b |
fixed×1multiplicity×1random_uniform×12witness state:
|
14 |
| union_bag library UNION ALL collapsed into OR — bag semantics lost | qwen2.5-coder:7b |
multiplicity×1nullable×1ties×1witness state:
|
3 |
| union_bag library UNION ALL collapsed into OR — bag semantics lost | qwen2.5:7b |
multiplicity×1nullable×1ties×1witness state:
|
3 |
| union_bag library UNION ALL collapsed into OR — bag semantics lost | phi3.5:3.8b |
multiplicity×1nullable×1ties×1witness state:
|
3 |
| union_bag library UNION ALL collapsed into OR — bag semantics lost | deepseek-coder-v2:latest |
multiplicity×1nullable×1random_uniform×9ties×1witness state:
|
12 |
| zero_fee library aggregate condition replaced by a row condition | qwen2.5-coder:7b |
multiplicity×1random_uniform×6ties×1zero_fee×1witness state:
|
9 |
| zero_fee library aggregate condition replaced by a row condition | llama3.2:3b |
boundary×1fixed×1multiplicity×1random_uniform×14zero_fee×1witness state:
|
18 |
| zero_fee library aggregate condition replaced by a row condition | qwen2.5:3b |
boundary×1fixed×1multiplicity×1random_uniform×15ties×1zero_fee×1witness state:
|
20 |
The other 129 evaluated families had no refuting state for any model: rewrites there were either preserved on the finite state bank or the model returned the reference unchanged. That is evidence of absence on this bank, not a proof of equivalence. All three schemas are represented in this table.
How to read these numbers
Outcomes are about evidence, not about “model quality” in the abstract
✕Refuted
At least one frozen database state makes the candidate return a different result than the reference query. The state, both result sets and a replay flag are stored in data/evidence/.
One counterexample is enough: a rewrite is wrong if any allowed state breaks it.
✓Preserved
The candidate differs from the reference SQL, and no state in the finite bank broke it. This is not a proof of equivalence — the bank is finite and cannot cover all possible databases.
Read it as “no evidence against on this bank”, never as “correct rewrite”.
–No rewrite
The model returned the reference query unchanged. That is neither a success nor a failure: there was no rewrite to validate.
Unsupported candidates and execution errors are separate outcomes and are reported explicitly, never folded into “no rewrite”.
ΣWhy checks beat shares
A pass rate hides the counterexample: one broken state among 5,712 checks is still a wrong rewrite. That is why this table shows refuting checks (629, 444, 415, 141, 111, 81) over executed checks (5,712), not percentages alone. Refuted share is shown for scale; the counterexample count is the evidence.
Every check is one executable state; the count is directly auditable in the data.
Data & links
Everything is public and recomputable from the repository
- Repositorygithub.com/ykvlev/sql-rewrite-evidence-bench
- Summary tablesdocs/tables.md — regenerated from evidence
- Dataset cardFormat, integrity rules and known limitations
- ReproduceChecks, hashes and evaluation protocol
- Zenodo DOIDOI assigned at release — placeholder, no DOI yet
- Hugging Face datasetCard prepared in docs/HF_CARD.md; hosting pending release