Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service - ACL Anthology

Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service

Hyeonmin Ha, Jihye Lee, Wookje Han, Byung-Gon Chun


Abstract
Recently, many companies have been providing the capabilities of large language models as services. These Language-Model-as-a-Service (LMaaS) offerings support a variety of user tasks through in-context learning from prompts, which include instructions and demonstrations of the task. However, for users, manually crafting prompts or running automatic prompt tuning methods themselves can be demanding. Despite these challenges, LMaaS providers do not offer automatic prompt engineering methods as part of their services. One of the major obstacles to deploying them on an LMaaS is the heavy computational costs associated with automatic prompt engineering methods. These methods are typically designed to iterate through tens of thousands of examples, which impose unaffordable overheads for LMaaS providers. In this paper, we introduce MetaL-Prompt, a novel lightweight automatic prompt generation method for LMaaS. MetaL-Prompt meta-trains a prompt generation model (PGM) to enable robust learning by the language model from the contexts created by the generated prompts (i.e., in-context learning). Thanks to our meta-learning approach, a PGM can generate prompts for unseen tasks without requiring additional training for those specific tasks. Furthermore, the PGM can generate prompts with a single forward pass, significantly reducing computational costs compared to previous methods. We evaluate MetaL-Prompt on a range of unseen tasks and find that it improves performance by up to 19.4% in terms of mean F1 score on QA datasets compared to the state-of-the-art baseline P-tuning, with limited computational cost.
Anthology ID:
2023.findings-emnlp.159
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2023
Month:
December
Year:
2023
Address:
Singapore
Editors:
Houda Bouamor, Juan Pino, Kalika Bali
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
2433–2445
Language:
URL:
https://aclanthology.org/2023.findings-emnlp.159
DOI:
10.18653/v1/2023.findings-emnlp.159
Bibkey:
Cite (ACL):
Hyeonmin Ha, Jihye Lee, Wookje Han, and Byung-Gon Chun. 2023. Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 2433–2445, Singapore. Association for Computational Linguistics.
Cite (Informal):
Meta-Learning of Prompt Generation for Lightweight Prompt Engineering on Language-Model-as-a-Service (Ha et al., Findings 2023)
Copy Citation:
PDF:
https://aclanthology.org/2023.findings-emnlp.159.pdf