Born a Transformer – Always a Transformer? On the Effect of Pretraining on Architectural Abilities - Jobanputra, Mayank; Veitsman, Yana; Sarrof, Yash; Bakalova, Aleksandra; Demberg, Vera; Pavlick, Ellie; Hahn, Michael (Advances in Neural Information Processing Systems, 2025)