DeCoOp: Robust Prompt Tuning
with Out-of-Distribution Detection

Zhi Zhou, Ming Yang, Jiang-Xin Shi, Lan-Zhe Guo‡, Yu-Feng Li‡
National Key Laboratory for Novel Software Technology, Nanjing University, China
‡Corresponding Author
zhouz@lamda.nju.edu.cn
Paper Code

Abstract

Vision-language models (VLMs), such as CLIP, have demonstrated impressive zero-shot capabilities for various downstream tasks. Their performance can be further enhanced through few-shot prompt tuning methods. However, current studies evaluate the performance of learned prompts separately on base and new classes. This evaluation lacks practicality for real-world applications since downstream tasks cannot determine whether the data belongs to base or new classes in advance. In this paper, we explore a problem setting called Open-world Prompt Tuning (OPT), which involves tuning prompts on base classes and evaluating on a combination of base and new classes. By introducing Decomposed Prompt Tuning framework (DePt), we theoretically demonstrate that OPT can be solved by incorporating out-of-distribution detection into prompt tuning, thereby enhancing the base-to-new discriminability. Based on DePt, we present a novel prompt tuning approach, namely, Decomposed Context Optimization (DeCoOp), which introduces new-class detectors and sub-classifiers to further enhance the base-class and new-class discriminability. Experimental results on 11 benchmark datasets validate the effectiveness of DePt and demonstrate that DeCoOp outperforms state-of-the-art methods, providing a significant 2% average accuracy improvement.

OPT Problem Setting

OPT Problem Setting

Figure: Overall illustation of OPT problem setting.

In this paper, we examine a problem setting known as Open-world Prompt Tuning (OPT), which focuses on tuning prompts for base classes and evaluating their performance on a combination of base and new classes. This setting allows for a comprehensive evaluation of the discriminabilities among the base-class, base-to-new, and new-class categories. We demonstrate that the accuracy in OPT does not align consistently with the previous H metric, as shown in the figure in our paper, indicating the need for a new evaluation metric for OPT.

DeCoOp Approach

DeCoOp Approach

Figure: Overall illustation of DeCoOp approach.

We propose the Decomposed Context Optimization (DeCoOp) approach to solve the OPT problem setting. DeCoOp integrates out-of-distribution (OOD) detection into prompt tuning, introducing new-class detectors to enhance the discriminability between the base and new classes. Additionally, DeCoOp employs sub-classifiers to further enhance the discriminability within the base class, thereby improving the performance of the base-class data. The original prompts are retained to ensure the robust performance of the new-class data. To address the issue of not having knowledge of the new-class data during training, we introduce an ensemble strategy to train the DeCoOp approach. The experimental results demonstrate that our DeCoOp approach surpasses state-of-the-art methods, effectively solving the OPT problem setting.

BibTeX

@inproceedings{zhou24decoop,
    author       = {Zhi Zhou and Ming Yang and Jiang-Xin Shi and Lan-Zhe Guo and Yu-Feng Li},
    title        = {DeCoOp: Robust Prompt Tuning with Out-of-Distribution Detection},
    booktitle    = {Proceedings of the 41st International Conference on Machine Learning},
    year         = {2024}
}