Qwen3-4B Candidate-Independent Decision Model

This is a trained Qwen3-4B decision model for scoring a variable-sized set of textual candidates. It uses candidate-independent block-causal attention:

• the state and question form a shared causal prefix;
• each candidate can attend to the shared prefix and its own tokens;
• candidates cannot attend to one another; and
• every candidate block restarts at the same logical position.

This prevents the model from using candidate serialization order as a shortcut while preserving causal computation inside each candidate. The model was trained for decision scoring on Open-Jev-style examples and produces a probability distribution over the displayed candidates.

This model is presented in the paper Permutation-Robust Decision Modeling with Candidate-Independent Block-Causal Attention.

Model checkpoint: Guy-Amit/qwen3-4b-ci-decision-4096-poc

Clone the code

Clone the project and enter the standalone model directory:

git clone https://github.com/guyAmit/ci-decision-models.git
cd ci-decision-models/ci-decision-models

Install the dependencies. On a GPU machine, install a CUDA-compatible PyTorch build first if necessary.

python -m pip install -r requirements.txt

Load the model

The checkpoint is downloaded from the Hugging Face Hub. It contains the Qwen3 backbone, tokenizer, and trained candidate-independent decision head.

import torch
from huggingface_hub import snapshot_download

from modeling_candidate_independent_qwen import CandidateIndependentQwen3

device = "cuda" if torch.cuda.is_available() else "cpu"
checkpoint = snapshot_download(
    repo_id="Guy-Amit/qwen3-4b-ci-decision-4096-poc",
    repo_type="model",
)
model, tokenizer = CandidateIndependentQwen3.from_checkpoint(
    checkpoint,
    device=device,
)
model.eval()

The same operation is available from the command line:

python load_qwen3_4b.py --device cuda

Serialization tokens

Each decision is serialized with the following special tokens:

<state> ... </state>
<question> ... </question>
<decision> <options>
<option> ... </option>
<option> ... </option>
...
</options>

<decision> is the shared query representation. The final token of each <option> ... </option> block is used as that candidate's representation. The model scores those candidate representations and applies a softmax over the available options.

Run one toy example

The following example uses the repository's real serializer and runs one decision through the downloaded model. It prints the serialized token sequence so the attention structure is visible, followed by the probability assigned to each candidate.

Save this as toy_example.py in ci-decision-models/ci-decision-models/, or run it directly with python.

import torch
from huggingface_hub import snapshot_download

from data_candidate_independent import CandidateIndependentCollator
from modeling_candidate_independent_qwen import CandidateIndependentQwen3


REPO_ID = "Guy-Amit/qwen3-4b-ci-decision-4096-poc"
device = "cuda" if torch.cuda.is_available() else "cpu"

checkpoint = snapshot_download(repo_id=REPO_ID, repo_type="model")
model, tokenizer = CandidateIndependentQwen3.from_checkpoint(
    checkpoint,
    device=device,
)
model.eval()

example = {
    "id": "toy-weather-1",
    "state": "The forecast says there is a high chance of rain this afternoon.",
    "question": "What should the person take before leaving home?",
    "options": [
        {"id": "umbrella", "text": "An umbrella"},
        {"id": "sunglasses", "text": "Sunglasses"},
        {"id": "swimsuit", "text": "A swimsuit"},
    ],
    "target": [1.0, 0.0, 0.0],
}

collator = CandidateIndependentCollator(tokenizer, max_length=1024)
batch = collator([example])

print("Serialized tokens:")
print(tokenizer.convert_ids_to_tokens(batch["input_ids"][0].tolist()))

model_inputs = {
    key: value.to(device)
    for key, value in batch.items()
    if torch.is_tensor(value)
}
with torch.inference_mode():
    output = model(**model_inputs)

probabilities = output.log_probs[0].exp().cpu().tolist()
print("
Candidate probabilities:")
for option, probability in zip(example["options"], probabilities):
    print(f"{option['text']}: {probability:.4f}")

For the example above, the output is a probability distribution over the three options. The model is not being prompted to generate text; it is scoring the candidate set supplied in the structured example.