Sparse Fine-Tuning of Transformers for Generative Tasks

Wei Chen, Jingxi Yu, Zichen Miao, Qiang Qiu
Purdue University
Paper arXiv Code (Coming soon...)
Our method represents the target concept using a learned dictionary

Building on a pre-trained DiT, our method represents the target concept using a learned dictionary.

Abstract

Fine-tuning pre-trained models is effective for adapting these models to downstream tasks. However, in existing fine-tuning methods, the updated representations are formed as a dense combination of modified parameters, making it challenging to interpret their contributions and understand how the model adapts to new tasks.

In this work, we introduce a fine-tuning framework inspired by sparse coding, where fine-tuned features are represented as a sparse combination of basic elements, i.e., feature dictionary atoms. The feature dictionary atoms function as fundamental building blocks of the representation, and tuning atoms allows for seamless adaptation to downstream tasks. Sparse coefficients then serve as indicators of atom importance, identifying the contribution of each atom to the updated representation. Leveraging the atom selection capability of sparse coefficients, we demonstrate that our method enhances image editing performance by improving text alignment through the removal of unimportant feature dictionary atoms.

Represent the target concept using the learned dictionary atoms

Approach: Our method leverages sparse coding and enforces high sparsity in adapted feature, ensuring that the output feature is constructed using only a few learned dictionary atoms.

Study of Dictionary Atoms

Effect of dictionary atoms on generated images

We visualize how dictionary atoms influence image generation. Use the slider below to adjust the number of atoms involved and observe how they contribute to forming the target concept.

Interpolate start reference image.

Pre-trained Model

Loading...
Interpolation end reference image.

Target Concept

Interpolate start reference image.

Pre-trained Model

Loading...
Interpolation end reference image.

Target Concept


Effect of individual dictionary atoms

Certain atoms exhibit distinct effects. For instance, atom 137 in layer 17 (L17[137]) introduces a yellow tint to the cat's face, while atom 139 in layer 3 (L3[139]) alters the cat's pose.

Represent the target concept using the learned dictionary atoms

Applications

Image editing

Building on our previous observation that adjusting different atoms influences the generated results, we explore leveraging this property for image editing.

Represent the target concept using the learned dictionary atoms

Adjust editing level

The density of involved atoms affects the level of image editing results.

Represent the target concept using the learned dictionary atoms

Concept and style merge

Since concepts or styles are captured by a small number of atoms, combining them becomes a natural way to generate images that reflect both the intended concept and style.

Represent the target concept using the learned dictionary atoms

BibTeX

@article{chen2025sparse,
  author    = {Wei Chen, Jingxi Yu, Zichen Miao, and Qiang Qiu},
  title     = {Sparse Fine-Tuning of Transformers for Generative Tasks},
  journal   = {ICCV},
  year      = {2025},
}