Automated Interaction Design for Intelligent Information Systems

Robin R. Penner, Ph.D. (robin@iterativity.com)
Iterativity, Inc. 118 E. 26^th St., Minneapolis, MN, 55404

The Automated Interaction Designer (AID) framework provides a ready-to-apply information system infrastructure, with an architecture that is explicitly designed to support distributed interactions between humans, information systems, and collaborative mixed-initiative agents. We have been investigating methods for supporting the human decision-maker who requires access to a full range of automation levels and a full spectrum of mixed-initiative control over complex semi-automated systems. Our recent investigations center on the area of human command and control of groups of intelligent machines, with the aim of reducing the number of humans required, resulting in the development of operational software for the DARPA MICA program.

The AID research addresses two important problems that stand in the way of producing good decision and control user interfaces. First, existing command and control systems do not represent the situation as a whole in ways that capture the semantic relationships between the situation components. This makes it difficult to present unified situational pictures to the decision maker. Adequate situation awareness is a cornerstone of decision-making, and provides the basis for the formulation of the command concept and control inputs by the human decision-maker. Second, user interfaces to current control systems do not allow the user to fully interact with the system at varying levels of detail, with the required flexibility, situational responsiveness, and adherence to good design principles that are necessary for an effective command dialog between human and automation. Any human command intent will be ineffective if it cannot be conveyed to the hardware and software systems that will implement the concept. In addition, without access to the full range of system behaviors, and without adequate and responsive interaction design, the human will be unable to perform activities that are required at various levels of the control hierarchy. Not only will the automation be unable to understand the human, it will be impossible for the human to understand the actions of automated components or know how and when to intervene.

AID (Penner & Steinmetz, 2002a, 2002b, 2003) demonstrates, in the domain of military operations planning and control, the use of a compositional model-based situation representation to provide dynamic interactions and user interfaces to cognitively support the warfighter. AID contains a semantically rich situation representation that automatically fuses information from multiple sources, and allows human users to interact with multiple intelligent automata with variable levels of autonomy. In addition, AID also dynamically creates, displays, and modifies all the user interfaces that are required by each user, based on user needs and the situational context. All aspects of the user interface are dynamically created on demand, specialized for the situation, the specific user, and the interface device they choose to use. Because all designs and interface implementations are based on consistent models of how best to meet user needs at all levels, the user interfaces are intrinsically easy to use and always incorporate the state of the art in interaction and user interface design, while fully supporting the dialog between human and machines.

Object-oriented models provide the knowledge bases for the multiple agents that are responsible for various aspects of situation representation and interaction and presentation design in AID. A domain model, under control of a situation agent, fuses the individual entities and data that are present in the battlespace into a coherent situation, providing the semantic basis for interaction design. A hierarchical, task-driven interaction model, managed by an interaction agent, contains the design knowledge required for automatic composition of appropriate tasks and abstract interactions, allowing dynamic application of the principles of usability engineering to support the design of the interactions between people and systems. A presentation model, driven by a presentation agent, possesses knowledge about how to select and format concrete user interfaces to express interaction designs, allowing hardware and operating system independence, and a clear separation between abstract interactions and concrete user interfaces.

The figure to the right shows the sequence of operations that provides dynamically designed user interfaces to situations. First, the set of unmet needs possessed by the human user in the situation causes AID to be invoked. The interaction agent determines which types of views onto the situation are necessary to meet the user's needs, then instantiates each type of view. The views compose themselves hierarchically by selecting the appropriate contents to represent the information needed. When the interaction is fully designed, the presentation agent is called to turn the interaction design into a user interface, as in the example screen shot below.

The AID approach demonstrates the usefulness of productive model-based reasoning for both situation representation and user interface design. Because the situation representation is dynamic, compositional, and hierarchical, a fully specified semantic structure is available at all times. Because the interaction design is based on a hierarchical situation model, the commander is provided with a user interface that allows situation assessment at various levels of detail. Because the interaction design is based on the commander's current needs, the most important interactions are immediately available. However, the interaction design

is also driven by the commander's capabilities, so all possible interactions are also easily accessible, providing the commander the ability to effortlessly traverse hierarchical levels of command. Because the interaction and presentation designs are productive and compositional, and their models contain rules of good design at all levels, the user interfaces require minimal training and are consistent, easy to use, and effective.

Penner, R. and Steinmetz, E. (2002a) Model-based automation of the design of user interfaces to digital control systems. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans. Vol 32, No. 1, January, pp 41-49.

Penner, R. and Steinmetz, E. (2002b) DIGBE: Online model-based design automation. Kolski and Vanderdonckt, eds., Computer-Aided Design of User Interfaces III, Kluwer, pp 179-192.

Penner, R. and Steinmetz, E. (2003) Implementation of automated interaction design with collaborative models. Interacting with Computers, Vol. 15, pp 367-385

Automated Interaction Design for Intelligent Information Systems Robin R. Penner, Ph.D. (robin@iterativity.com) Iterativity, Inc. 118 E. 26^th St., Minneapolis, MN, 55404 The *Automated Interaction Designer* (AID) framework provides a ready-to-apply information system infrastructure, with an architecture that is explicitly designed to support distributed interactions between humans, information systems, and collaborative mixed-initiative agents. We have been investigating methods for supporting the human decision-maker who requires access to a full range of automation levels and a full spectrum of mixed-initiative control over complex semi-automated systems. Our recent investigations center on the area of human command and control of groups of intelligent machines, with the aim of reducing the number of humans required, resulting in the development of operational software for the DARPA MICA program. The AID research addresses two important problems that stand in the way of producing good decision and control user interfaces. First, existing command and control systems do not represent the situation as a whole in ways that capture the semantic relationships between the situation components. This makes it difficult to present unified situational pictures to the decision maker. Adequate situation awareness is a cornerstone of decision-making, and provides the basis for the formulation of the command concept and control inputs by the human decision-maker. Second, user interfaces to current control systems do not allow the user to fully interact with the system at varying levels of detail, with the required flexibility, situational responsiveness, and adherence to good design principles that are necessary for an effective command dialog between human and automation. Any human command intent will be ineffective if it cannot be conveyed to the hardware and software systems that will implement the concept. In addition, without access to the full range of system behaviors, and without adequate and responsive interaction design, the human will be unable to perform activities that are required at various levels of the control hierarchy. Not only will the automation be unable to understand the human, it will be impossible for the human to understand the actions of automated components or know how and when to intervene. AID (Penner & Steinmetz, 2002a, 2002b, 2003) demonstrates, in the domain of military operations planning and control, the use of a compositional model-based situation representation to provide dynamic interactions and user interfaces to cognitively support the warfighter. AID contains a semantically rich situation representation that automatically fuses information from multiple sources, and allows human users to interact with multiple intelligent automata with variable levels of autonomy. In addition, AID also dynamically creates, displays, and modifies all the user interfaces that are required by each user, based on user needs and the situational context. All aspects of the user interface are dynamically created on demand, specialized for the situation, the specific user, and the interface device they choose to use. Because all designs and interface implementations are based on consistent models of how best to meet user needs at all levels, the user interfaces are intrinsically easy to use and always incorporate the state of the art in interaction and user interface design, while fully supporting the dialog between human and machines. Object-oriented models provide the knowledge bases for the multiple agents that are responsible for various aspects of situation representation and interaction and presentation design in AID. A domain model, under control of a situation agent, fuses the individual entities and data that are present in the battlespace into a coherent situation, providing the semantic basis for interaction design. A hierarchical, task-driven interaction model, managed by an interaction agent, contains the design knowledge required for automatic composition of appropriate tasks and abstract interactions, allowing dynamic application of the principles of usability engineering to support the design of the interactions between people and systems. A presentation model, driven by a presentation agent, possesses knowledge about how to select and format concrete user interfaces to express interaction designs, allowing hardware and operating system independence, and a clear separation between abstract interactions and concrete user interfaces.
The figure to the right shows the sequence of operations that provides dynamically designed user interfaces to situations. First, the set of unmet needs possessed by the human user in the situation causes AID to be invoked. The interaction agent determines which types of views onto the situation are necessary to meet the user's needs, then instantiates each type of view. The views compose themselves hierarchically by selecting the appropriate contents to represent the information needed. When the interaction is fully designed, the presentation agent is called to turn the interaction design into a user interface, as in the example screen shot below.
	The AID approach demonstrates the usefulness of productive model-based reasoning for both situation representation and user interface design. Because the situation representation is dynamic, compositional, and hierarchical, a fully specified semantic structure is available at all times. Because the interaction design is based on a hierarchical situation model, the commander is provided with a user interface that allows situation assessment at various levels of detail. Because the interaction design is based on the commander's current needs, the most important interactions are immediately available. However, the interaction design
is also driven by the commander's capabilities, so all possible interactions are also easily accessible, providing the commander the ability to effortlessly traverse hierarchical levels of command. Because the interaction and presentation designs are productive and compositional, and their models contain rules of good design at all levels, the user interfaces require minimal training and are consistent, easy to use, and effective.
Penner, R. and Steinmetz, E. (2002a) Model-based automation of the design of user interfaces to digital control systems. IEEE Transactions on Systems, Man and Cybernetics, Part A: Systems and Humans. Vol 32, No. 1, January, pp 41-49. Penner, R. and Steinmetz, E. (2002b) DIGBE: Online model-based design automation. Kolski and Vanderdonckt, eds., Computer-Aided Design of User Interfaces III, Kluwer, pp 179-192. Penner, R. and Steinmetz, E. (2003) Implementation of automated interaction design with collaborative models. Interacting with Computers, Vol. 15, pp 367-385