Details

Autor(en) / Beteiligte

• Wang, Wenhui
• Bao, Hangbo
• Dong, Li
• Bjorck, Johan
• Peng, Zhiliang
• Liu, Qiang
• Aggarwal, Kriti
• Mohammed, Owais Khan
• Singhal, Saksham
• Som, Subhojit
• Wei, Furu

Titel

Image as a Foreign Language: BEIT Pretraining for Vision and Vision-Language Tasks

Ist Teil von

• 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, p.19175-19186

Ort / Verlag

IEEE

Erscheinungsjahr

2023

Quelle

IEEE Xplore

Beschreibungen/Notizen

• A big convergence of language, vision, and multimodal pretraining is emerging. In this work, we introduce a general-purpose multimodal foundation model BEIT-3, which achieves excellent transfer performance on both vision and vision-language tasks. Specifically, we advance the big convergence from three aspects: backbone architecture, pretraining task, and model scaling up. We use Multiway Transformers for general-purpose modeling, where the modular architecture enables both deep fusion and modality-specific encoding. Based on the shared backbone, we perform masked "language" modeling on images (Imglish), texts (English), and image-text pairs ("parallel sentences") in a unified manner. Experimental results show that BEIT-3 obtains remarkable performance on object detection (COCO), semantic segmentation (ADE20K), image classification (ImageNet), visual reasoning (NLVR2), visual question answering (VQAv2), image captioning (COCO), and cross-modal retrieval (Flickr30K, COCO).