The dynamic nature of cyber threats presents significant challenges for modern defense, as sophisticated adversaries continuously adapt their strategies to evade detection and compromise valuable systems. Effective defense against these evolving threats requires multiagent interaction, where human defenders must coordinate with both other humans and AI systems to mount comprehensive responses. However, current approaches fail to adequately model the cognitive mechanisms underlying multiagent interactions in these complex environments. Without computational models of how humans adapt, collaborate, and make decisions in cybersecurity contexts, we cannot build multiagent defense systems that leverage the full potential of human and AI.

This thesis focuses on building computational cognitive models and cog- nitive agents for multiagent interaction in cyber defense, including designing adversarial cognitive agents (Chapter 3), modeling human decision-making in multi-defender interaction (Chapter 4), and designing human-like AI agents that can work with humans as a team (Chapter 5).

First, I investigate human behavior in cybersecurity at the individual level and build adversarial cognitive agents that capture human-like adaptivity in cyber attack, which pre-sent greater challenges to defenders than deterministic strategies. My findings show that cognitive attackers driven by Instance-Based Learning Theory can learn effective strategies that are more challenging for both human and autonomous defenders to counter than optimal but predictable attack patterns.

Second, I explore cognitive mechanisms that enable effective decision-making in multi-defender interactions. In cybersecurity, multiple defenders can share sensitive information and collaborate on threat response, however, their willingness to do so could impact the security posture of all connected defenders. I develop a novel computational model for interdependent human decision-making and investigate its validity in multi-defender interaction setting. The model incorporates three key cognitive mechanisms: dynamic prosociality, which adjusts how individuals value others' outcomes based on expectation-reality discrepancies; category learning, which efficiently organizes social experiences into behavioral prototypes; and contrast effects, which sharpen distinctions between these behavioral categories.

Finally, I examine the integration of human and AI decision-making in team defense scenarios where humans and AI collaboratively protect computer networks. I designed an AI agent that learns from experience to approximate human-like decision processes. Through empirical studies in semi-supervisory frameworks, I demonstrate that the human-like AI agent significantly enhances team performance and efficiency in cybersecurity operations compared to heuristic or random agents.

221 pages

Thesis Committee:
Fei Fang (Co-chair)
Cleotilde Gonzalez (Co-chair)
Christian Lebiere
Prashanth Rajivan (University of Washington)
Tiffany Bao (Arizona State University)

Nicolas Christin, Head, Software and Societal Systems Department
Martial Hebert, Dean, School of Computer Science

Return to: SCS Technical Report Collection
School of Computer Science

This page maintained by reports@cs.cmu.edu