Skip to content

Honor start_date/end_date and release_version on every loader path - #154

Open
AUTHENSOR wants to merge 1 commit into
LiveCodeBench:mainfrom
AUTHENSOR:fix/thread-selection-flags
Open

AUTHENSOR wants to merge 1 commit into
LiveCodeBench:mainfrom
AUTHENSOR:fix/thread-selection-flags

Conversation

@AUTHENSOR

Copy link
Copy Markdown

Summary

--start_date and --end_date select the contamination window and --release_version pins the dataset revision, but all three are honored only on the default code_generation_lite path. On --not_fast (the README's documented way to run the original benchmark with the full private test suites), load_code_generation_dataset_not_fast has no date parameters at all and never forwards release_version to load_dataset, so the request silently resolves to whatever the Hub repo's default config currently is and every problem is loaded, including problems outside the requested window. selfrepair drops the dates; testoutputprediction and codeexecution accept release_version and never use it. No warning or error mentions any of this, so a run's contamination-free property is decided by the union of loaded problems, not by the flags on the command line.

Fixes #153.

Changes

  • load_code_generation_dataset_not_fast now takes start_date/end_date, forwards version_tag=release_version, and applies the same contest-date filtering as the fast path.
  • build_prompt_benchmark passes the date flags on the not_fast and selfrepair branches.
  • load_test_prediction_dataset and load_code_execution_dataset forward version_tag=release_version.

Verification

The repo has no test suite, so the change ships with a self-contained verifier (https://gist.github.com/AUTHENSOR/6583bcdd3863d7b54b6db3884f9933f1) that drives the real build_prompt_benchmark with the HuggingFace load_dataset boundary stubbed to record its arguments and return four synthetic problems (two inside 2024-01-01..2024-06-01, one before, one after). Five arms:

fast_control:        window applied, version_tag=release_v5
not_fast:            window applied (2/4 in window), version_tag=release_v5 forwarded
selfrepair:          window applied (2/4)
testoutputprediction: version_tag=release_v5 forwarded
codeexecution:       version_tag=release_v5 forwarded

On current main the same verifier fails at the not_fast arm exactly as the issue describes: 4/4 problems loaded, out-of-window problems present, and load_dataset called with no version_tag ({'split': 'test'} only).

--not_fast silently ignored --start_date/--end_date (no date parameters
at all) and never forwarded --release_version to load_dataset, so the
original-benchmark path loaded the full dataset at the Hub's default
revision with no signal. selfrepair dropped the dates;
testoutputprediction and codeexecution accepted release_version but
never used it.

Thread the flags through every loader: not_fast now forwards
version_tag and applies the same contest-date window as the fast path,
selfrepair passes its dates through, and the test-generation and
execution loaders forward version_tag.

Fixes LiveCodeBench#153
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

--not_fast silently ignores --start_date/--end_date and --release_version; other scenarios ignore the date flags too

1 participant