Pareto Multi-Task Learning
通过限定子区域实现 Pareto MTL
NeurIPS
An Image is Worth 32 Tokens for Reconstruction and Generation
非结构化的一维离散 Visual Tokens 生成策略
Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction
自回归图像生成: Multi-scale Quantization & Next-scale Prediction
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
多尺度 Transformer, 探究非 Subword Tokenizer 的可能性
SpaceByte: Towards Deleting Tokenization from Large Language Modeling
探究非 Subword Tokenizer 的可能性
Universal Prompt Tuning for Graph Neural Networks
图上特征 prompt 等价各异 graph prompt
Base of RoPE Bounds Context Length
讨论 RoPE base 对于相似 Tokens 感知能力的影响
Transformers need glasses! Information Over-Squashing in Language Tasks
LLM Representational Collapse
MICROADAM: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence
MicroAdam, 通过梯度稀疏化以及 error compensation 实现轻量的优化器
EMR-MERGING: Tuning-Free High-Performance Model Merging
EMR-MERGING, 多任务权重合并技术