Details

Autor(en) / Beteiligte

• Luo, Shiya
• Chen, Defang
• Wang, Can

Titel

Customizing Synthetic Data for Data-Free Student Learning

Ist Teil von

• 2023 IEEE International Conference on Multimedia and Expo (ICME), 2023, p.1817-1822

Ort / Verlag

IEEE

Erscheinungsjahr

2023

Quelle

IEL

Beschreibungen/Notizen

• Data-free knowledge distillation (DFKD) aims to obtain a lightweight student model without original training data. Existing works generally synthesize data from the pretrained teacher model to replace the original training data for student learning. To more effectively train the student model, the synthetic data shall be customized to the current student learning ability. However, this is ignored in the existing DFKD methods and thus negatively affects the student training. To address this issue, we propose Customizing Synthetic Data for Data-Free Student Learning (CSD) in this paper, which achieves adaptive data synthesis using a self-supervised augmented auxiliary task to estimate the student learning ability. That is, data synthesis is dynamically adjusted to enlarge the cross entropy between the labels and the predictions from the self-supervised augmented task, thus generating the hard samples for the student model. The experiments on various datasets and teacher-student models show the effectiveness of our proposed method. Code is available at: https://github.com/luoshiya/CSD