Skip to content

Fix Pix2StructTextAttention init using hidden_size instead of d_kv - #47558

Merged
Cyrilvallez merged 3 commits into
huggingface:mainfrom
yuvrajnode:fix-pix2struct-attention-init-dkv
Sep 3, 2026
Merged

Cyrilvallez merged 3 commits into
huggingface:mainfrom
yuvrajnode:fix-pix2struct-attention-init-dkv

Conversation

@yuvrajnode

@yuvrajnode yuvrajnode commented Jul 26, 2026 •

Copy link
Copy Markdown
Contributor

CPU CI GPU run-slow

What does this PR do?

Pix2StructTextAttention's weight initialization in _init_weights computes key_value_proj_dim with a fallback that doesn't match the pattern used everywhere else in the same block:

key_value_proj_dim = (
    self.config.text_config.d_kv if isinstance(self.config, Pix2StructConfig) else self.config.hidden_size
)

Every other line in this elif isinstance(module, Pix2StructTextAttention): block mirrors the same leaf attribute name in both branches:

hidden_size = self.config.text_config.hidden_size if isinstance(self.config, Pix2StructConfig) else self.config.hidden_size
n_heads = self.config.text_config.num_heads if isinstance(self.config, Pix2StructConfig) else self.config.num_heads

but key_value_proj_dim falls back to self.config.hidden_size instead of self.config.d_kv, even though Pix2StructTextConfig.d_kv is a distinct field (default 64, versus hidden_size which defaults much higher).

#Impact

When Pix2StructTextModel (or Pix2StructTextAttention) is instantiated standalone — i.e. with a plain Pix2StructTextConfig rather than wrapped in the composite Pix2StructConfig the isinstance check fails, and key_value_proj_dim silently becomes hidden_size instead of d_kv. This gives the wrong initialization std for the query, key, value, and output projection weights.

I verified this numerically: with hidden_size=64, d_kv=16, the observed init std for query.weight matched factor * (hidden_size * hidden_size) ** -0.5 (the buggy formula) rather than factor * (hidden_size * d_kv) ** -0.5 (correct, matching the Pix2StructConfig-wrapped path).

@Cyrilvallez @Rocketknight1

@github-actions

Copy link
Copy Markdown
Contributor

[For maintainers] Suggested jobs to run (before merge)

run-slow: pix2struct

@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 30330356624:2
Result: success | Jobs: 5 | Tests: 650 | Failures: 0 | Duration: 38s

@Cyrilvallez Cyrilvallez left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed thanks

@Cyrilvallez
Cyrilvallez merged commit c66311b into huggingface:main Sep 3, 2026
34 checks passed
sbucaille pushed a commit to sbucaille/transformers that referenced this pull request Sep 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants