Details

Autor(en) / Beteiligte

• Douillard, Arthur
• Rame, Alexandre
• Couairon, Guillaume
• Cord, Matthieu

Titel

DyTox: Transformers for Continual Learning with DYnamic TOken eXpansion

Ist Teil von

• 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, p.9275-9285

Ort / Verlag

IEEE

Erscheinungsjahr

2022

Link zum Volltext

• IEEE_Xplore

Quelle

IEEE Electronic Library Online

Beschreibungen/Notizen

• Deep network architectures struggle to continually learn new tasks without forgetting the previous tasks. A recent trend indicates that dynamic architectures based on an ex-pansion of the parameters can reduce catastrophic forget-ting efficiently in continual learning. However, existing approaches often require a task identifier at test-time, need complex tuning to balance the growing number of parameters, and barely share any information across tasks. As a result, they struggle to scale to a large number of tasks without significant overhead. In this paper, we propose a transformer architecture based on a dedicated encoder/decoder framework. Critically, the encoder and decoder are shared among all tasks. Through a dynamic expansion of special tokens, we specialize each forward of our decoder network on a task distribution. Our strategy scales to a large number of tasks while having neg-ligible memory and time overheads due to strict control of the expansion of the parameters. Moreover, this efficient strategy doesn't need any hyperparameter tuning to control the network's expansion. Our model reaches excellent results on CIFAR100 and state-of-the-art performances on the large-scale ImageNet100 and ImageNet100 while having fewer parameters than concurrent dynamic frameworks. 1 1 Code is released at https://github.com/arthurdouillard/dytox.