Trajectory Consistency Distillation
Improved Latent Consistency Distillation by Semi-Linear Consistency Function with Trajectory Mapping
* Equal contribution
A Solemn Statement Regarding the Plagiarism Allegations
We regret to hear about the serious accusations from the CTM team.
We sadly found out our CTM paper (ICLR24) was plagiarized by TCD! It's unbelievable😢—they not only stole our idea of trajectory consistency but also committed "verbatim plagiarism," literally copying our proofs word for word! Please help me spread this. pic.twitter.com/aR6pRjhj5X
— Dongjun Kim (@gimdong58085414) March 25, 2024
Before this post, we already had several rounds of communication with CTM's authors. We shall proceed to elucidate the situation here.
We regret to hear about the serious accusations from the CTM team @gimdong58085414. I shall proceed to elucidate the situation and make an archive here. We already have several rounds of communication with CTM's authors. https://t.co/BKn3w1jXuh
— Michael (@Merci0318) March 26, 2024
- In the first arXiv version (which is accused of plagiarism), we have provided citations and discussion in A. Related Works:
- In the first arXiv version, we have indicated in D.3 Proof of Theorem 4.2
In this section, our derivation mainly borrows the proof from (Kim et al., 2023; Chen et al., 2022).
and we have never intended to claim credits.
As we have mentioned in our email, we would like to extend a formal apology to the CTM authors for the clearly inadequate level of referencing in our paper. We will provide more credits in the revised manuscript. - In the updated second arXiv version, we have expanded our discussion to elucidate the relationship with CTM. Additionally, we have removed some proofs that were previously included for completeness.
- CTM and TCD are different from motivation, method to experiments. TCD is founded on the principles of the Latent Consistency Model (LCM), aimed to design an effective consistency function by utilizing the exponential integrators.
- The experimental results also cannot be obtained from any type of CTM algorithm.
- Here we provide a simple method to check: use our sampler here to sample the checkpoint CTM released, or vice versa.
- CTM also provided a training script. We welcome anyone to reproduce the experiments on SDXL based on the CTM algorithm.
We believe the assertion of plagiarism is not only severe but also detrimental to the academic integrity of the involved parties. We earnestly hope that everyone involved gains a more comprehensive understanding of this matter.
Abstract
Latent Consistency Model (LCM) extends the Consistency Model to the latent space and leverages the guided consistency distillation technique to achieve impressive performance in accelerating text-to-image synthesis. However, we observed that LCM struggles to generate images with both clarity and detailed intricacy.
Comparison between TCD and other state-of-the-art methods. TCD delivers exceptional results in terms of both quality and speed, completely surpassing LCM. Notably, LCM experiences a notable decline in quality at high NFEs. In contrast, TCD maintains superior generative quality at high NFEs, even exceeding the performance of DPM-Solver++(2S) with origin SDXL.
To address this limitation, we initially delve into and elucidate the underlying causes. Our investigation identifies that the primary issue stems from errors in three distinct areas. Consequently, we introduce Trajectory Consistency Distillation (TCD), which encompasses trajectory consistency function (TCF) and strategic stochastic sampling (SSS) .
Training process, wherein the TCF expands the boundary conditions to an arbitrary timestep of s, thereby reducing the theoretical upper limit of error.
Sampling process, where SSS significantly reduces accumulated error through the iterative traversal with the stochastic parameter compared to the multistep consistency sampling
The trajectory consistency function diminishes the distillation errors by broadening the scope of the self-consistency boundary condition with a semi-linear consistency function and endowing the TCD with the ability to accurately trace the entire trajectory of the Probability Flow ODE.
Experiments demonstrate that TCD not only significantly enhances image quality at low NFEs but also yields more detailed results compared to the teacher model at high NFEs.
Better than Teacher w/o Additional Supervision
TCD maintains superior generative quality at both low NFEs and high NFEs, even exceeding the performance of DPM-Solver++(2S) with origin SDXL. It is worth noting that there is no additional discriminator or LPIPS supervision included during training.
We demonstrate some examples at 20 NFEs below.
Comparision with SDXL
Flexible NFEs
The NFEs for TCD sampling can be varied at will (compared with Turbo series), without adversely affecting the quality of the results (compared with LCMs).
We compare the performance of TCD and LCM at increasing NFEs.
Comparision with LCM at increasing NFEs
Freely Change the Detailing
During inference, the level of detail in the image can be simply modified by adjusing one hyper-parameter gamma. This option does not require the introduction of any additional trainable 32parameters.
Comparision with LCM at different 𝛾
Versatility
TCD can be adapted to various SDXL-based expansions and plugins in the community, for instance, LoRA, ControlNet, IP Adapter, and other base models, e.g. Animagine XL.
Related and Concurrent Work
- Luo S, Tan Y, Huang L, et al. Latent consistency models: Synthesizing high-resolution images with few-step inference. arXiv preprint arXiv:2310.04378, 2023.
- Luo S, Tan Y, Patil S, et al. LCM-LoRA: A universal stable-diffusion acceleration module. arXiv preprint arXiv:2311.05556, 2023.
- Lu C, Zhou Y, Bao F, et al. DPM-Solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. Advances in Neural Information Processing Systems, 2022, 35: 5775-5787.
- Lu C, Zhou Y, Bao F, et al. DPM-solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
- Zhang Q, Chen Y. Fast sampling of diffusion models with exponential integrator. ICLR 2023, Kigali, Rwanda, May 1-5, 2023.
- Kim D, Lai C H, Liao W H, et al. Consistency Trajectory Models: Learning Probability Flow ODE Trajectory of Diffusion. ICLR 2024.
Citation
@misc{zheng2024trajectory,
title = {Trajectory Consistency Distillation},
author = {Jianbin Zheng and Minghui Hu and Zhongyi Fan and Chaoyue Wang and Changxing Ding and Dacheng Tao and Tat-Jen Cham},
archivePrefix = {arXiv},
eprint = {2402.19159},
year = {2024},
primaryClass = {cs.CV}
}
Acknowledgements
The website style was inspired from DreamFusion.