Details

Autor(en) / Beteiligte

• Kwon, Eunji
• Song, Haena
• Park, Jihye
• Kang, Seokhyeong

Titel

Mobile Accelerator Exploiting Sparsity of Multi-Heads, Lines, and Blocks in Transformers in Computer Vision

Ist Teil von

• 2023 Design, Automation & Test in Europe Conference & Exhibition (DATE), 2023, p.1-6

Ort / Verlag

EDAA

Erscheinungsjahr

2023

Quelle

IEEE Electronic Library (IEL)

Beschreibungen/Notizen

• It is difficult to employ transformer models for computer vision in mobile devices due to their memory- and computation-intensive properties. Accordingly, there is ongoing research on various methods for compressing transformer models, such as pruning. However, general computing platforms such as central processing units (CPUs) and graphics processing units (GPUs) are not energy-efficient to accelerate the pruned model due to their structured sparsity. This paper proposes a low-power accelerator for transformers with various sizes of structured sparsity induced by pruning with different granularity. In this study, we can accelerate a transformer that has been pruned in a head-wise, line-wise, or block-wise manner. We developed a head scheduling algorithm to support head-wise skip operations and resolve the processing engine (PE) load imbalance problem caused by different number of operations in one head. Moreover, we implemented a sparse general matrix-to-matrix multiplication (sparse GEMM) module that supports line-wise and block-wise skipping. As a result, when compared with a mobile GPU and mobile CPU respectively, our proposed accelerator achieved 6.1\times and 13.6\times improvements in energy efficiency for the detection transformer (DETR) model and achieved approximately 2.6\times and 7.9\times improvements in the energy efficiency on average for the vision transformer (ViT) models.