Recent advances in large language models have furthered the development of agentic systems: pipelines that interleave planning, execution, and iterative refinement. The following thesis demonstrates the design, implementation, and evaluation of such agentic systems so they can be leveraged in underutilized ways, from productivity tasks to refining kernel code. First, we propose Web LLM Agent, which showcases a general-purpose Chrome extension agent that integrates LLMs with diverse web APIs. Beyond constructing a web agent, we aimed to extend agentic systems to the more specialized and impactful domains, such as creating kernels through an LLM Kernel Agent. This agent embodies an iterative decision-making agent that generates, benchmarks, and refines kernels, illustrating the core principles of agent architecture, feedback loops, and sequential optimization. Findings will show that there is a lack of thorough benchmarking and evaluation metrics for kernels, making iterative, generative improvement difficult. The third component of the following paper focuses on creating a lightweight, extensible benchmarking suite, FlashInfer Bench, which resolves this issue. The suite captures execution traces to systematically evaluate and compare low-level kernel implementations for performance and correctness in model inference workloads. Thus, FlashInfer Bench proposes a self-improving, community-driven platform for developing and deploying hardware-aware, high-performance kernels. Together, these contributions emphasize the central importance of agent design, tool orchestration, and integrated feedback loops in unlocking autonomous, high-performance agents.

41 pages

Thesis Committee:
Tianqi Chen (Advisor)
Zhihao Jia

Srinivasan Seshan, Head, Computer Science Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu