Language models have improved by orders of magnitude with the recent emergence of Transformer-based Large Language Models (LLMs). LLMs have demonstrated their ability to generate "natural" code that is highly similar to code written by professional developers. However, the software engineering process involves much more than writing code: modern software evolves and requires continuous maintenance, such as debugging, or transpilation. For a LLM to assist in the software engineering process, it is important to build tools around a LLM to enable its ability to provide support for software evolution. In this thesis proposal, we propose mechanisms to improve the utility of LLMs for software evolution by using and combining LLMs with prior APR and program verification techniques. Specifically, we build LLM-based software engineering tools for fault localization, program repair, and program transpilation.

My thesis statement is: Program analysis complements large language models (LLMs) for software maintenance by providing structured software signals that include bidirectional code context, formal verification of code properties, and dataflow properties. These signals can be integrated with LLMs beyond autoregressive generation. Explicitly integrating analysis-derived signals with LLMs consistently outperforms either pure learning-based or pure analysis-based approaches across key software engineering tasks in real-world repositories.

To support this statement, we make the following contributions.

      I first propose and evaluate a bidirectional fine-tuning technique that enables a previously left-to-right LLM to locate and rank faulty lines of code. We built the LLM-based fault localization technique without depending on preciously written tests, and so the tool can also detect runtime security vulnerabilities.

       I further study a LLM's ability for program repair by combining the LLM's intermediate entropy values with traditional program repair techniques. In particular, we used LLM entropy values to improve three stages of program repair: fault localization, patch testing efficiency, and plausible patch ranking. Finally, we created a tool for fully automated Rust function transpilation using LLMs and verification harnesses. To support our claims, we evaluated all our LLM-based software evolution tools on real world bugs, security vulnerabilities, and target transpilation repositories.

      I extend our study of LLM-based fault localization by adding mechanisms for multi-task LLM instruction-tuning, and evaluating the model on real-world security vulnerabilities in large repositories. Specifically, the completed fault localizer only detects faults on a line level, and does not take into account various nuances of security vulnerability explanations. LLMs have the capability of training on multiple objectives, including both recognizing the vulnerable lines of code, and the explanation of exploits occurring from a type of vulnerability. The completed vulnerability detector attempts to detect vulnerabilities that span across different files across a larger repository.

106 pages

Thesis Committee:
Claire Le Goues (Co-Chair)
Ruben Martins (Co-Chair)
Vincent Hellendoorn (Google DeepMind)
Daniel Kroening (Amazon Web Services)

Nicolas Christin, Head, Software and Societal Systems Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu