Dynamic Chunking for End-to-End Hierarchical Sequence Modeling
符号序列的自动切分, 探究非 Subword Tokenizer 的可能性
Tokenization
MEGABYTE: Predicting Million-byte Sequences with Multiscale Transformers
多尺度 Transformer, 探究非 Subword Tokenizer 的可能性
SpaceByte: Towards Deleting Tokenization from Large Language Modeling
探究非 Subword Tokenizer 的可能性