Nandan Kumar Jha

@TheRegularizer

PhD @nyuniversity | LLM pre-training, representation geometry & optimization | Spectral scaling laws | Prev: @IITHyderabad @Seagate

Brooklyn, NY

nankj.com

Joined May 2018

186 Following

68 Followers

Replies Reposts Media

Pinned
Nandan Kumar Jha
@TheRegularizer
Aug 16
Great to see our Spectral Scaling Laws work discussed from the perspective of FFN/MoE design! This work was started out of curiosity, to know how wide should an FFN be, and how do we know whether that width is actually being utilized in LLMs? And it was based on the fact that,
Andy
@prompt_Tunes
Jun 8
paper reading thread! i've been exploring spectral scaling laws for FFNs and MOEs recently and hence my thoughts on one of the papers that i found really interesting in the way the authors recast width selection as a spectral utilization optimization problem. (1/n)
2
Nandan Kumar Jha
@TheRegularizer
Jul 8
Author here, thanks for reading & posting! On behavioral analysis: we think the connection to the tail associative-memory result is plausible, they observed Muon storing tail facts better; our work present a representational counterpart, TAIL hard-rank scaling exponent β=1.02
Rosinality
@rosinality
May 22
arxiv.org/abs/2605.21803 Scaling of effective rank of post activation over FFN hidden dimensions, per token frequency, is quite different for Adam and Muon even when loss is matched. And this interacts with architectural choices like RoPE. Maybe related to tail knowledge behavior
1
Nandan Kumar Jha
@TheRegularizer
Jul 8
🚨 New paper! Optimizer-Induced Spectral Scaling Laws We investigate how the optimizer geometry modulates representation geometry and affect latent space utilization across token-regimes. Holding architecture and training-data fixed, changing optimizer changes the scaling
1
Nandan Kumar Jha
@TheRegularizer
Jan 26
Our paper NerVE has been accepted to ICLR 2026! 🎉🎉🎉 We study how Feed-Forward Networks (FFNs) in language models organize information in high-dimensional latent space, and show how nonlinearities reinject variance across eigenmodes---strongly modulated by optimizer (e.g.,
Nandan Kumar Jha
@TheRegularizer
Mar 6, 2025
A thought-provoking take on what it truly takes to drive scientific breakthroughs!
Thomas Wolf
@Thom_Wolf
Mar 6, 2025
I shared a controversial take the other day at an event and I decided to write it down in a longer format: I’m afraid AI won't give us a "compressed 21st century". The "compressed 21st century" comes from Dario's "Machine of Loving Grace" and if you haven’t read it, you probably