The advent of large language models has sparked a surge of interest in both academia and industry, inspiring the exploration and development of AI Agents for software engineering. However, the enthusiasm surrounding impressive demos in this field belies the reality that creating these agents is more challenging than anticipated. In this talk, I will delve into the key challenges and innovative solutions we have identified in architecting these systems, evaluating their effectiveness, and maintaining them.

First, I will shed light on the design objectives needed to construct a collaborative debugging agent. I will introduce the “investigate and respond” agentic pattern, which we have applied to enhance GitHub Copilot in Visual Studio. I will further demonstrate how we are broadening this concept to develop task-centric agents, highlighting their efficacy in tasks such as fixing build errors and GitHub issues. In the latter part of the talk, I will present a technique for evaluating Human-AI conversations, and discuss how we can augment the performance of AI agents by capitalizing on experiential learnings derived from past trials via a process of meta-reflection.