Details

Autor(en) / Beteiligte

• Cormode, Graham
• Markov, Igor L.

Titel

Federated Calibration and Evaluation of Binary Classifiers

Ist Teil von

• Proceedings of the VLDB Endowment, 2023-07, Vol.16 (11), p.3253-3265

Erscheinungsjahr

2023

Quelle

ACM Digital Library

Beschreibungen/Notizen

• We address two major obstacles to practical deployment of AI-based models on distributed private data. Whether a model was trained by a federation of cooperating clients or trained centrally, (1) the output scores must be calibrated, and (2) performance metrics must be evaluated --- all without assembling labels in one place. In particular, we show how to perform calibration and compute the standard metrics of precision, recall, accuracy and ROC-AUC in the federated setting under three privacy models ( i ) secure aggregation, ( ii ) distributed differential privacy, ( iii ) local differential privacy. Our theorems and experiments clarify tradeoffs between privacy, accuracy, and data efficiency. They also help decide if a given application has sufficient data to support federated calibration and evaluation.