Securing AI Coding Assistants

Code Large Language Models (LLMs) like GitHub Copilot, Amazon CodeWhisperer, and Code Llama have revolutionized software development, boosting productivity for millions of developers. By 2028, it is estimated that 75% of enterprise software engineers will rely on AI Coding Assistants. However, Code LLMs raise significant security concerns, with studies indicating that 40% of programs generated by GitHub Copilot are vulnerable. Thus, there is an urgent need to ensure the safety of Code LLMs.In this lecture, I will begin by reviewing the traditional methods used to evaluate the functional correctness and security of code produced by LLMs, pointing out the inherent limitations of these approaches. I will then introduce a new benchmark and new metrics we've developed to more accurately assess the correctness and security of Code LLMs. I will delve into how Code LLMs generate code and our research on the security of the state-of-the-art models. Finally, I will discuss future research directions and challenges of how to generate secure code.