Within-subject
Full-FTLinear Probe
Pretraining Data Scaling
RQ: Under a fixed model architecture, how does pretraining data scale affect downstream task performance?
Downstream Performance
Downstream performance under within-subject and cross-subject settings for Full-FT and Linear Probe across different pretraining data scales.
Cross-subject
Full-FTLinear Probe
Adaptation Dynamics
Relative parameter displacement and model output representational change induced by Full-FT. Layer-wise representational change across layers and pretraining data scales.
Parameter & output change
30k hours · 0.037 displacement · 0.901 change
Layer-wise representational change
30k hours · Layer 22 · 0.583
00.75