From Learning through Labels to Learning through Language

Over the past decades, machine learning has primarily relied on labeled data, with success often depending on the availability of vast, high-quality annotations and the assumption that test conditions mirror training conditions. In contrast, humans learn efficiently from conceptual explanations, instructions, rules, and contextual understanding. With advancements in large language models, AI systems can now understand descriptions and follow instructions, paving the way for a paradigm shift. Inspired by this, we investigate how to empower AI agents to learn from natural language narratives and multimodal interactions with humans, making them more adaptable to rapidly changing environments.
This talk explores how teaching machines through language can enable AI systems to gain human trust and enhance their interpretability, robustness, and ability to learn new concepts. I will highlight our journey in developing vision-language models capable of detecting unseen objects through rich natural language descriptions. Additionally, I will discuss techniques for guiding the behavior of language models and text-to-image models using language, and how we generate human-readable features using vision-language models. Finally, I will conclude the talk by discussing future directions and potential challenges in empowering models to learn through language.