← Back to README

• 6.1 Positional Encoding Variants
• 6.2 YaRN / NTK / Position Interpolation
• 6.3 Length Extrapolation & Inference-Time Context Extension

1. RoFormer: Enhanced Transformer with Rotary Position Embedding. Jianlin Su, Yu Lu, Shengfeng Pan, Ahmed Murtadha, Bo Wen, Yunfeng Liu. Arxiv 2021.

2. KERPLE: Kernelized Relative Positional Embedding for Length Extrapolation. Ta-Chung Chi, Ting-Han Fan, Peter J. Ramadge, Alexander I. Rudnicky. Arxiv 2022.

3. Randomized Positional Encodings Boost Length Generalization of Transformers. Anian Ruoss, Grégoire Delétang, Tim Genewein, Jordi Grau-Moya, Róbert Csordás, Mehdi Bennani, Shane Legg, Joel Veness. ACL 2023.

4. The Impact of Positional Encoding on Length Generalization in Transformers. Amirhossein Kazemnejad, Inkit Padhi, Karthikeyan Natesan Ramamurthy, Payel Das, Siva Reddy. Arxiv 2023.

5. Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation. Zhenyu He, Guhao Feng, Shengjie Luo, Kai Yang, Di He, Jingjing Xu, Zhi Zhang, Hongxia Yang, Liwei Wang. ICML 2024.

6. Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding. Zhenyu Zhang, Runjin Chen, Shiwei Liu, Zhewei Yao, Olatunji Ruwase, Beidi Chen, Xiaoxia Wu, Zhangyang Wang. Arxiv 2024.

7. DAPE: Data-Adaptive Positional Encoding for Length Extrapolation. Chuanyang Zheng, Yihang Gao, Han Shi, Minbin Huang, Jingyao Li, Jing Xiong, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li. NeurIPS 2024.

8. Contextual Position Encoding: Learning to Count What's Important. Olga Golovneva, Tianlu Wang, Jason Weston, Sainbayar Sukhbaatar. Arxiv 2024.

9. Position Coupling: Improving Length Generalization of Arithmetic Transformers Using Task Structure. Hanseul Cho, Jaeyoung Cha, Pranjal Awasthi, Srinadh Bhojanapalli, Anupam Gupta, Chulhee Yun. NeurIPS 2024.

10. An Efficient Recipe for Long Context Extension via Middle-Focused Positional Encoding. Tong Wu, Yanpeng Zhao, Zilong Zheng. NeurIPS 2024.

11. 3D-RPE: Enhancing Long-Context Modeling Through 3D Rotary Position Encoding. Xindian Ma, Wenyuan Liu, Peng Zhang, Nan Xu. Arxiv 2024.

12. DAPE V2: Process Attention Score as Feature Map for Length Extrapolation. Chuanyang Zheng, Yihang Gao, Han Shi, Jing Xiong, Jiankai Sun, Jingyao Li, Minbin Huang, Xiaozhe Ren, Michael Ng, Xin Jiang, Zhenguo Li, Yu Li. Arxiv 2024.

13. HoPE (NoDecay): A Novel Positional Encoding Without Long-Term Decay for Enhanced Context Awareness and Extrapolation. Yuhan Chen, Ang Lv, Jian Luan, Bin Wang, Wei Liu. Arxiv 2024.

14. Circuit Complexity Bounds for RoPE-based Transformer Architecture. Bo Chen, Xiaoyu Li, Yingyu Liang, Jiangxuan Long, Zhenmei Shi, Zhao Song. Arxiv 2024.

15. DINT Transformer. Yueyang Cang, Yuhang Liu, Xiaoteng Zhang, Erlu Zhao, Li Shi. Arxiv 2025.

16. The Rotary Position Embedding May Cause Dimension Inefficiency in Attention Heads for Long-Distance Retrieval. Ting-Rui Chiang, Dani Yogatama. Arxiv 2025.

17. Forgetting Transformer: Softmax Attention with a Forget Gate. Zhixuan Lin, Evgenii Nikishin, Xu Owen He, Aaron Courville. ICLR 2025.

18. Layer-Specific Scaling of Positional Encodings for Superior Long-Context Modeling. Zhenghua Wang, Yiran Ding, Changze Lv, Zhibo Xu, Tianlong Li, Tianyuan Shi, Xiaoqing Zheng, Xuanjing Huang. Arxiv 2025.

19. SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling. Krishna C. Puvvada, Faisal Ladhak, Santiago Akle Serrano, Cheng-Ping Hsieh, Shantanu Acharya, Somshubra Majumdar, Fei Jia, Samuel Kriman, Simeng Sun, Dima Rekesh, Boris Ginsburg. Arxiv 2025.

20. Effective Length Extrapolation via Dimension-Wise Positional Embeddings Manipulation. Yi Lu, Wanxu Zhao, Xin Zhou, Chenxin An, Chenglong Wang, Shuo Li, Yuming Yang, Jun Zhao, Tao Ji, Tao Gui, Qi Zhang, Xuanjing Huang. Arxiv 2025.

21. Mitigating Posterior Salience Attenuation in Long-Context LLMs with Positional Contrastive Decoding. Zikai Xiao, Ziyang Wang, Wen Ma, Yan Zhang, Wei Shen, Yan Wang, Luqi Gong, Zuozhu Liu. Arxiv 2025.

22. Bayesian Attention Mechanism: A Probabilistic Framework for Positional Encoding and Context Length Extrapolation. Arthur S. Bianchessi, Rodrigo C. Barros, Lucas S. Kupssinskü Arxiv 2025.

23. HoPE (Hyperbolic): Hyperbolic Rotary Positional Encoding for Stable Long-Range Dependency Modeling in Large Language Models. Chang Dai, Hongyu Shan, Mingyang Song, Di Liang. Arxiv 2025.

24. Positional Encoding via Token-Aware Phase Attention. Yu, Wang, Sheng Shen, Rémi Munos, Hongyuan Zhan, Yuandong Tian. Arxiv 2025.

25. RePo: Language Models with Context Re-Positioning. Huayang Li, Tianyu Zhao, Richard Sproat. Arxiv 2025.

26. Length Generalization of Causal Transformers without Position Encoding. Jie Wang, Tao Ji, Yuanbin Wu, Hang Yan, Tao Gui, Qi Zhang, Xuanjing Huang, Xiaoling Wang. Arxiv 2024.

27. Rope to Nope and Back Again: A New Hybrid Attention Strategy. Bowen Yang, Bharat Venkitesh, Dwarak Talupuru, Hangyu Lin, David Cairuz, Phil Blunsom, Acyr Locatelli. Arxiv 2025.

28. Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings (DroPE). Yoav Gelberg, Koshi Eguchi, Takuya Akiba, Edoardo Cetin. Arxiv 2025.

29. Extensible Embedding: A Flexible Multipler For LLM's Context Length. Ninglu Shao, Shitao Xiao, Zheng Liu, Peitian Zhang. Arxiv 2024.

30. Base of RoPE Bounds Context Length. Xin Men, Mingyu Xu, Bingning Wang, Qingyu Zhang, Hongyu Lin, Xianpei Han, Weipeng Chen. Arxiv 2024.

31. Periodic RoPE for Infinite Context LLMs. Simin Huo. Arxiv 2026.

32. Disentangling the Expressivity of RoPE. Selim Jerad, Anej Svete, Jiaoda Li, Ryan Cotterell. Arxiv 2026.

33. Anti-Periodic Positional Encoding: Möbius Boundary Conditions Make In-Context Retrieval Reliable. Ji Ho Bae. Arxiv 2026.

1. Extending Context Window of Large Language Models via Positional Interpolation. Shouyuan Chen, Sherman Wong, Liangjian Chen, Yuandong Tian. Arxiv 2023.

2. YaRN: Efficient Context Window Extension of Large Language Models. Bowen Peng, Jeffrey Quesnelle, Honglu Fan, Enrico Shippole. Arxiv 2023.

3. Scaling Laws of RoPE-based Extrapolation. Xiaoran Liu, Hang Yan, Shuo Zhang, Chenxin An, Xipeng Qiu, Dahua Lin. Arxiv 2023.

4. LongRoPE: Extending LLM ContextWindow Beyond 2 Million Tokens. Yiran Ding, Li Lyna Zhang, Chengruidong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, Mao Yang. Arxiv 2024.

5. CLEX: Continuous Length Extrapolation for Large Language Models. Guanzheng Chen, Xin Li, Zaiqiao Meng, Shangsong Liang, Lidong Bing. Arxiv 2023.

6. Resonance RoPE: Improving Context Length Generalization of Large Language Models. Suyuchen Wang, Ivan Kobyzev, Peng Lu, Mehdi Rezagholizadeh, Bang Liu. Arxiv 2024.

7. A Training-Free Length Extrapolation Approach for LLMs: Greedy Attention Logit Interpolation (GALI). Yan Li, Tianyi Zhang, Zechuan Li, Soyeon Caren Han. Arxiv 2025.

8. LongRoPE2: Near-Lossless LLM Context Window Scaling. Ning Shang, Li Lyna Zhang, Siyuan Wang, Gaokai Zhang, Gilsinia Lopez, Fan Yang, Weizhu Chen, Mao Yang. Arxiv 2025.

1. Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolation. Ofir Press, Noah A. Smith, Mike Lewis. ICLR 2022.

2. Dissecting Transformer Length Extrapolation via the Lens of Receptive Field Analysis. Ta-Chung Chi, Ting-Han Fan, Alexander I. Rudnicky, Peter J. Ramadge. ACL 2023.

3. A Length-Extrapolatable Transformer. Yutao Sun, Li Dong, Barun Patra, Shuming Ma, Shaohan Huang, Alon Benhaim, Vishrav Chaudhary, Xia Song, Furu Wei. ACL 2023.

4. Exploring Transformer Extrapolation. Zhen Qin, Yiran Zhong, Hui Deng. Arxiv 2023.

5. LM-Infinite: Simple On-the-Fly Length Generalization for Large Language Models. Chi Han, Qifan Wang, Wenhan Xiong, Yu Chen, Heng Ji, Sinong Wang. Arxiv 2023.

6. Attention Alignment and Flexible Positional Embeddings Improve Transformer Length Extrapolation. Ta-Chung Chi,Ting-Han Fan,Alexander I. Rudnicky. Arxiv 2023.

7. CoCA: Fusing position embedding with Collinear Constrained Attention for fine-tuning free context window extending. Shiyi Zhu, Jing Ye, Wei Jiang, Qi Zhang, Yifan Wu, Jianguo Li. Arxiv 2023.

8. LLM Maybe LongLM: Self-Extend LLM Context Window Without Tuning. Hongye Jin, Xiaotian Han, Jingfeng Yang, Zhimeng Jiang, Zirui Liu, Chia-Yuan Chang, Huiyuan Chen, Xia Hu. Arxiv 2024.

9. Transformers Can Achieve Length Generalization But Not Robustly. Yongchao Zhou, Uri Alon, Xinyun Chen, Xuezhi Wang, Rishabh Agarwal, Denny Zhou. Arxiv 2024.

10. Can't Remember Details in Long Documents? You Need Some R&R. Devanshu Agrawal, Shang Gao, Martin Gajek. Arxiv 2024.

11. InfLLM: Unveiling the Intrinsic Capacity of LLMs for Understanding Extremely Long Sequences with Training-Free Memory. Chaojun Xiao, Pengle Zhang, Xu Han, Guangxuan Xiao, Yankai Lin, Zhengyan Zhang, Zhiyuan Liu, Song Han, Maosong Sun. Arxiv 2024.

12. Naive Bayes-based Context Extension for Large Language Models. Jianlin Su, Murtadha Ahmed, Wenbo, Luo Ao, Mingren Zhu, Yunfeng Liu. Arxiv 2024.

13. Explicitly Encoding Structural Symmetry is Key to Length Generalization in Arithmetic Tasks. Mahdi Sabbaghi, George Pappas, Hamed Hassani, Surbhi Goel. Arxiv 2024.

14. Mixture of In-Context Experts Enhance LLMs' Long Context Awareness. Hongzhan Lin, Ang Lv, Yuhan Chen, Chen Zhu, Yang Song, Hengshu Zhu, Rui Yan. Arxiv 2024.

15. Efficient Solutions For An Intriguing Failure of LLMs: Long Context Window Does Not Mean LLMs Can Analyze Long Sequences Flawlessly. Peyman Hosseini, Ignacio Castro, Iacopo Ghinassi, Matthew Purver. Arxiv 2024.

16. Extending Context Window of Large Language Models from a Distributional Perspective. Yingsheng Wu, Yuxuan Gu, Xiaocheng Feng, Weihong Zhong, Dongliang Xu, Qing Yang, Hongtao Liu, Bing Qin. Arxiv 2024.

17. Why Does the Effective Context Length of LLMs Fall Short?. Chenxin An, Jun Zhang, Ming Zhong, Lei Li, Shansan Gong, Yao Luo, Jingjing Xu, Lingpeng Kong. Arxiv 2024.

18. Two are better than one: Context window extension with multi-grained self-injection. Wei Han, Pan Zhou, Soujanya Poria, Shuicheng Yan. Arxiv 2024.

19. What is Wrong with Perplexity for Long-context Language Modeling?. Lizhe Fang, Yifei Wang, Zhaoyang Liu, Chenheng Zhang, Stefanie Jegelka, Jinyang Gao, Bolin Ding, Yisen Wang. Arxiv 2024.

20. Transformers Can Do Arithmetic with the Right Embeddings. Sean Michael McLeish, Arpit Bansal, Alex Stein, Neel Jain, John Kirchenbauer, Brian R. Bartoldson, Bhavya Kailkhura, Abhinav Bhatele, Jonas Geiping, Avi Schwarzschild, Tom Goldstein. NeurIPS 2024.

21. Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count. Hanseul Cho, Jaeyoung Cha, Srinadh Bhojanapalli, Chulhee Yun. Arxiv 2024.

22. Attention Entropy is a Key Factor: An Analysis of Parallel Context Encoding with Full-attention-based Pre-trained Language Models. Zhisong Zhang, Yan Wang, Xinting Huang, Tianqing Fang, Hongming Zhang, Chenlong Deng, Shuaiyi Li, Dong Yu. Arxiv 2024.

23. DCIS: Efficient Length Extrapolation of LLMs via Divide-and-Conquer Scaling Factor Search. Lei Yang, Shaoyang Xu, Deyi Xiong. Arxiv 2024.

24. Information Entropy Invariance: Enhancing Length Extrapolation in Attention Mechanisms. Kewei Li, Yanwen Kong, Yiping Xu, Lan Huang, Ruochi Zhang, Fengfeng Zhou. Arxiv 2025.

25. SEAL: Scaling to Emphasize Attention for Long-Context Retrieval. Changhun Lee, Jun-gyu Jin, Younghyun Cho, Eunhyeok Park. Arxiv 2025.

26. Unveiling Simplicities of Attention: Adaptive Long-Context Head Identification. Konstantin Donhauser, Charles Arnal, Mohammad Pezeshki, Vivien Cabannes, David Lopez-Paz, Kartik Ahuja. Arxiv 2025.

27. ParallelComp: Parallel Long-Context Compressor for Length Extrapolation. Jing Xiong, Jianghan Shen, Chuanyang Zheng, Zhongwei Wan, Chenyang Zhao, Chiwun Yang, Fanghua Ye, Hongxia Yang, Lingpeng Kong, Ngai Wong. ICML 2025.

28. SELF: Self-Extend the Context Length With Logistic Growth Function. Phat Thanh Dang, Saahil Thoppay, Wang Yang, Qifan Wang, Vipin Chaudhary, Xiaotian Han. Arxiv 2025.

29. Hierarchical Context Merging: Better Long Context Understanding for Pre-trained LLMs. Woomin Song, Seunghyuk Oh, Sangwoo Mo, Jaehyung Kim, Sukmin Yun, Jung-Woo Ha, Jinwoo Shin. ICLR 2024.

30. Compress, Gather, and Recompute: REFORMing Long-Context Processing in Transformers. Woomin Song, Sai Muralidhar Jayanthi, Srikanth Ronanki, Kanthashree Mysore Sathyendra, Jinwoo Shin, Aram Galstyan, Shubham Katiyar, Sravan Babu Bodapati. Arxiv 2025.

31. Causal Attention with Lookahead Keys. Zhuoqing Song, Peng Sun, Huizhuo Yuan, Quanquan Gu. Arxiv 2025.

32. XL3M: A Training-free Framework for LLM Length Extension Based on Segment-wise Inference. Shengnan Wang, Youhui Bai, Lin Zhang, Pingyi Zhou, Shixiong Zhao, Gong Zhang, Sen Wang, Renhai Chen, Hua Xu, Hongwei Sun. Arxiv 2024.

33. Soaring from 4K to 400K: Extending LLM's Context with Activation Beacon. Peitian Zhang, Zheng Liu, Shitao Xiao, Ninglu Shao, Qiwei Ye, Zhicheng Dou. Arxiv 2024.

34. Flexibly Scaling Large Language Models Contexts Through Extensible Tokenization. Ninglu Shao, Shitao Xiao, Zheng Liu, Peitian Zhang. Arxiv 2024.

35. Self-Guided Test-Time Training for Long-Context LLMs. Xinyu Zhu, Zhe Xu, Xiaohan Wei, Yunchen Pu, Fei Tian, Chonglin Sun, Kaushik Rangadurai, Hua Zhi, Frank Shyu, Sandeep Pandey, Luke Simon, Yu Meng, Xi Liu. Arxiv 2026.

36. Jet-Long: Efficient Long-Context Extension with Dynamic Bifocal RoPE. Haozhan Tang, Zerui Wang, Yuxian Gu, Song Han, Han Cai. Arxiv 2026.

37. EndPrompt: Efficient Long-Context Extension via Terminal Anchoring. Han Tian, Luxuan Chen, Xinran Chen, Rui Kong, Fang Wang, Jiamin Chen, Jinman Zhao, Yuchen Li, Jiashu Zhao, Shuaiqiang Wang, Haoyi Xiong, Dawei Yin. Arxiv 2026.

38. From Rigid to Dynamic: Entropy-Guided Adaptive Inference for Long-Context LLMs. Zhanchao Xu, Haoyang Li, Qingfa Xiao, Fei Teng, Chen Jason Zhang, Lei Chen, Qing Li. Arxiv 2026.