Distill the Best, Ignore the Rest: A Study in Latent Dataset Distillation on Core-Sets

Abstract

Latent dataset distillation, which exploits pre-trained generative priors, has gained significant interest in recent years because it can be applied agnostic to any distillation algorithm and addresses two significant limitations of classical distillation algorithms: cross-architecture generalization and high-resolution synthesis. However, existing approaches typically distill from the entire dataset, potentially including non-beneficial samples. We introduce a novel “Prune First, Distill After” framework that systematically prunes datasets via loss-based sampling prior to latent distillation. By leveraging pruning before classical distillation techniques and generative priors, we create a representative coreset that leads to enhanced generalization for unseen architectures - a significant challenge of current distillation methods. More specifically, our proposed framework significantly boosts distilled quality, achieving up to a 5.2 percentage points accuracy increase even with substantial dataset pruning, i.e., removing 80% of the original dataset prior to distillation. Overall, our experimental results highlight the advantages of our easy-sample prioritization and cross-architecture robustness, paving the way for more effective and high-quality dataset distillation.

Dr. Federico Raue

Senior Researcher

MSc. Tobias Nauen

Research assistant at the Artificial Intelligence Chair

I’m a researcher of artificial intelligence at DFKI and RPTU Kaiserslautern-Landau.

Prof. Dr. Prof. h.c. Andreas Dengel

Executive Director of DFKI, Chair of Artificial Intelligence (Department of Computer Science, TU Kaiserslautern)

add info