The videos from the event are available: [Day 1 Presentations, Day 2 Presentations]

From In the Loop to On the Loop

The workshop will include invited presentations with moderated discussions and facilitated sessions to generate new solutions to different aspects of the challenge problems. The workshop will cover 2 days of sessions, each looking at one of the aforementioned challenge problem areas. The workshop will bring together representatives from the top NSA/DHS Designated Centers of Academic Excellence in Information Assurance-Research for a series of focused meetings and technical exchanges to better understand the risks, challenges, and opportunities of the growing adoption of ML/AI in cyber security and the social-technical implications of such trends in future cyber operations. Participants will collaborate, share ideas, and present research identifying innovative and practical solutions to one or more of the challenge problems.


The increasing adoption of Machine Learning (ML) and Artificial Intelligence (AI) in cyber operations are revolutionizing the way we handle computer network defense. Multiple examples in both research and commercial applications are starting to show a path for the long-promised concept of autonomous cyber defense.

One issue, however, that is often overlooked is most practical computational or cyber-physical systems require some interface to humans either as end-users operators or the systems. Traditionally, the concepts of computer network defense relied primarily on providing some of level of visibility, understanding, and control of the computational infrastructure and potential threats to operators who would use numerous tools and ensure the secure and trusted execution of tasks enabled by the system. However, the fast-approaching reality of autonomous cyber defense has changed the way humans interface with the systems.

The role of the operator has been progressively transitioning from making individual response decisions to directing the types of response decisions made for different classes of incidents. This has been referred to as going from "in the loop" to "on the loop." Understanding how operators can best make this transition and best employ their creative and contextual reasoning capabilities to large-scale cybersecurity activities is not well understood.

The socio-technical infrastructure needed to ensure that the directors have timely, complete, correct, and trustworthy information needed to accurately guide cyber infrastructure has also not reached consensus in either component structure (what is required of the infrastructure to produce the information needed) or information content (how can the needed information properties be determined and communicated.)

Critical infrastructures are increasingly dependent on cyberspace, increasing the number of different areas that require safeguarding

Topics of Interest

The ability to provide direction in response to fast-moving and fast-changing threats requires addressing a number of challenging problems:

Perception, Deception, and Trust - this refers to what the person understands from the elements they perceive and how they can be deceived intentionally or unintentionally either by systems or by other people.

Teamwork - this component refers to the collaboration of people with systems and with other people. It includes shared models, theory of mind, collaborative sense-making and many related issues.

Co-evolution - this aspect deals with how attackers adapt to changes in system defense, and how defenders adapt to the attackers’ evolving strategies.

Identity - this component deals with the multi-dimensional aspect of human identity and how it can be used or misused in cyber security.

Cyber Analytics - this challenge addresses the measuring and representing the speed, scale, and accuracy of cybersecurity technologies and the data streams they operate. Creating intelligent analytics can support both enhanced automated capabilities and greater situational awareness for the operator.

Testing and Performance Measurement - this component is critical to the advancement of an integrated socio-technical cybersecurity infrastructure by ensuring accurate and repeatable tests for proposed systems and operation direction techniques.

Machine Learning/AI in Cyber and Teams - this component addresses how recent advances in machine learning and artificial intelligence technologies can be applied to cyber security data and processes to both improve performance and to expand capabilities. Many of these advances may be particularly applicable to addressing the integration of humans into the learning/reasoning processes.