FI Computational Methods and Data Science Journal Club
Bi-Weekly Speaker with Reception to follow
We invite you to FI Computational Methods and Data Science Journal Club on Thursday, August 5, 2021, at 3:00 pm, 162 5th Ave, 5th floor, Classroom. Please respond to the RSVP found here by no later than COB on Tuesday, August 3rd if you wish to attend.
We will need participants to RSVP bi-weekly per current COVID protocols and for headcount.
Flatiron Institute, 162 5th Avenue
Speaker: Kyunghyun Cho
Title: True Few-Shot Learning with Language Models
Abstract: Pretrained language models (LMs) perform well on many tasks even when learning from a few examples, but prior work uses many held-out examples to tune various aspects of learning, such as hyperparameters, training objectives, and natural language templates ("prompts"). Here, we evaluate the few-shot ability of LMs when such held-out examples are unavailable, a setting we call true few-shot learning. We test two model selection criteria, cross-validation, and minimum description length, for choosing LM prompts and hyperparameters in the true few-shot setting. On average, both marginally outperform random selection and greatly underperform selection based on held-out examples. Moreover, selection criteria often prefer models that perform significantly worse than randomly selected ones. We find similar results even when taking into account our uncertainty in a model's true performance during selection, as well as when varying the amount of computation and number of examples used for selection. Overall, our findings suggest that prior work significantly overestimated the true few-shot ability of LMs given the difficulty of few-shot model selection.
paper: https://arxiv.org/abs/2105.11447