Abstract
Proper design of human-computer interactions currently requires the application of specialized knowledge possessed by talented individuals, often responding to rapidly changing technology and functionality. As computing power and data become more distributed, static design of interactions for dynamic environments may not always produce usable interfaces between people and machines. We have designed a model-based, dynamic interaction design system called DIG (Dynamic Interaction Generation) and implemented it in a research prototype using the domain of digital building control systems. DIGBE is a three-tiered system modeling the domain data, the current interaction, and the device–specific presentation of the interaction. It responds to changes in the environment and creates, in real-time, an interaction tailored to the current user role, access level, and task. Each interaction is also specialized to match the nature of the information that is currently the focus of the interaction. Full automatic synthesis of any interface at any time may not yet be feasible, but DIGBE demonstrates that it is both possible and useful to automatically generate user interfaces. This may be especially true for complex systems domains with variable components but known task structures.
Introduction
A crucial determinant of the quality of human-computer collaboration within modern automated systems is the proper design of the “interaction-space” where the human and computer communicate. In general, proper interaction design is difficult to achieve, mainly because design requires the application of specialized, ever-changing knowledge possessed by talented individuals, responding to rapidly evolving technology and functionality. In addition, interaction design professionals often operate in environments where it is difficult to quantify procedures or results, or to even produce useful specifications, standards, or guidelines.
Even when performed optimally, static interaction design is often no longer sufficient to meet the complex task and information needs of current systems. Increases in computing power and the appearance of more ubiquitous, distributed capabilities will make the tasks and information that users need even more unpredictable, greatly increasing the difficulty of providing good interaction design. Given the non-deterministic nature of an ubiquitous computing environment with numerous computational and human agents, static design will no longer be sufficient to provide useful and usable interfaces between people and machines. It will be impossible to predict the interactions that will be required for a particular system and user, the information that will be necessary, or even the information that will be available. It will also be impossible to predict or control the hardware and software capabilities that will exist in local and distributed forms, or the experience and capabilities of the human and software participants.
In the particular area of mixed initiative control of complex distributed systems, other factors complicate the ability to deliver good user interfaces. Even within a domain like building management, the individual installations are very different from one another. Small buildings differ from large buildings, forced air systems differ from boiler systems, and automated security installations differ from buildings that are staffed with human security guards. The environment within which each system operates and the special parameters required for their installation and operation cannot be predicted in advance.
The variability of the system user also complicates the delivery of well-designed interactions to complex systems. In a large installation, for example, the roles of the various users may be well-specified and limited to one or two of the tasks required in the system. In a smaller installation, the tasks may all be assigned to a single user. In addition, the tasks and roles in the system must interact with various users’ access levels. In most systems, some users are allowed to change certain values, while other users with the same role, but lower access privileges, may only view these values without making changes.
This variability among systems in terms of configuration, criticality, tasks, user roles, and security levels drives up the costs of user interface development considerably. In traditional systems, each display that each user requires must be individually hand crafted. Appropriate visualization user interfaces for the specific plan or building must be created individually, at a significant cost. Hence, providing human usability over today's broad range of building, security, and industrial systems is a difficult and costly task for both the producers and the consumers of digital control systems [1].
We believe that the needs of users interacting with complex control systems can only be met with dynamic interaction design. As newer systems emerge, the unpredictability of tasks and information available will only become more important, overshadowing the difficulty of defining and applying dynamic design knowledge. In order to explore the possibility of automatically designing user interfaces, we have developed a model-based solution called Dynamic Interaction Generation (DIG) [2].
The DIG research prototype, DIGBE (DIG for Building Environments) [3], is an automated reasoning system composed of model-driven collaborative agents. When a human user asks for an interaction with a control system, DIGBE designs the user interface, presents it to the user on the user's selected device, and adapts it as the interaction progresses in real time. These user interfaces are consistent, well designed, and specialized to the user privileges, the roles of the interaction participants, the specific task, the objects of interest, and the values and types of data. DIGBE can also respond to the interaction device with which the user is currently interacting; the research implementation demonstrates the ability to generate automatically designed user interfaces for either a CRT or a handheld Palm.
Related Research
Researchers and practitioners have attempted to address the “design gap” problem in a number of ways. When graphical user interfaces became practical in the 1980s, emphasis on user interface management systems (UIMS) resulted in attempts to automate graphical user interface generation. In the mid-1990’s, industrial and university researchers generally concluded [4] that user interface generation is too difficult, because it depends on human knowledge of task structures and domain requirements. Many of these UIMS researchers chose to concentrate on approaches in which automated, model-based systems provide critique of static designs developed (mainly) by human designers [5].
Despite the lack of emphasis on automated generation, the related work on computer assisted design has had much to offer in the areas of user and task modeling, and also informs the issues of separation of processes between domains, interactions, and presentations. SAGE [6] generated 2D static presentations of relational data, and emphasized the importance of multi-dimensional representations of data and the effect of user goals on effective design. BOZ [7] took a task analytic approach to interface automation, included some rules of composition that responded to situational parameters, and separated interaction content from user interface format. IBIS [8] had a separate presentation component, and included representation knowledge of users, tasks, and contexts. Later work on IBIS included an emphasis on data characterization [9] and hierarchical decomposition of visual lexicons [10].
Other approaches to modeling and user interface development inform the current work. Investigations in the area of User Interface Design Environments (UIDE) includes modeling of user interface design knowledge [11], separation of design environments into multiple layers for the domain, the interactions, and the presentations [12], and declarative user interface design models [13]. Mecano [14] uses domain models to provide the underlying semantics for user interface design. Recently, Computer Aided Design of User Interfaces (CADUI) [15] has been suggested as a more efficient emphasis than automated generation. CADUI provides incremental design aiding through a task based (rather than a widget model based) mechanism, and includes an automatic presentation layer.
Driven by the high costs of delivering good user interfaces for each installation of a complex control system, some industrial practitioners have been developing pragmatic solutions to the problem. In the domain of manufacturing information systems, a generated user interface has been demonstrated [16] which combines user, task, and information modeling with templates to produce dynamic, adaptive interfaces for controlling automated manufacturing parameters. Our approach contains these elements, but also incorporates an adaptive model of the design of interactions for subtasks and interaction elements that can modify themselves for different users, data, or task situations.
Dynamic Interaction Generation
Although researchers in the field of intelligent user interfaces and user interface automation have not been completely enthusiastic about the chances for truly automated design, our work was motivated by the desperate need for situated interface design in the domain of mixed-initiative digital control.
The domains of building management and industrial process control are well understood, and domain models with information about data types and information roles are possible to construct. We were fortunate to have been involved in the development of a well-defined framework for evolvable graphical user interfaces to process control tasks [17,18]. This framework, based on abstracted principles of best practices user interface design (e.g. [19] and [20]), was prototyped and tested as a general user interface framework for complex control [17]. Both its usability and usefulness were established, making it an appropriate test case as the basis for interaction design automation. We reasoned that it should be possible to automate the design process that humans would use when applying this framework to build applications for a particular domain, set of users, and set of tasks.
DIGBE
DIGBE is an implementation of a set of modular automated reasoners that automatically and dynamically design and create user interfaces to large buildings and their associated control systems. It is a fully functional system for managing the basic tasks of building management, including configuration, monitoring and control of security and environmental systems, and management of users.
DIGBE converts a building management configuration database into a dynamic, adaptive user interface to building management functionality. It uses a modern digital control system as the source of real time operational data, and uses configuration files native to the selected platform for information about the objects in the domain.
With no user interface code from a building management system (BMS) application, DIGBE generates the required navigation, monitoring, and data modification interactions required for several different classes of BMS users. Figure 1 shows a screen shot of a DIGBE generated interaction for a technician for a large building environmental control system. Each user interface (UI) is presented as a coherent application, specialized to that user's role and access privileges when the user logs on. A heating and ventilation (HVAC) technician, for example, would see only heating and ventilation information, and have access to a full range of data and data modification capabilities. A security guard, on the other hand, has access to very different tasks and limited control over a different set of information. A user at a PC sees graphical navigation hierarchies and color-coding of status information, but this same information is shown as a text hierarchy with emphasis coding on a Palm.
DIGBE provides a well-designed, specialized interface for each user. It supports the collaboration between the user and the system, by:
- automatically designing an appropriate application shell and required task interactions,
- dynamically specializing this interface based on the current user and domain situation, and
- interactively presenting the user interface for the interactions that are required on the user's device.
Figure 1. DIGBE-Generated User Interface for HVAC Technician
Model Based Interaction Design
The heart of DIGBE's automated design capabilities is the knowledge that its Interaction Agent possesses about dynamic interaction design. This knowledge is contained in the Task and Interaction Model (TIM). The purpose of DIGBE’s TIM is to model the process of designing the interactions between the user and the data that are required by the current situation
The Task and Interaction Model is made up of five compositional levels, each a hierarchy of examples of types at that level:
- Applications, like Building Management, which are composed of tasks;
- Tasks, like Monitoring or Specifying, which are composed of subtasks;
- Sub-Tasks, like Plotting or Selection, which are composed of elements;
- Elements, like Discrete Set or Value Over Time, which are composed of primitives; and
- Primitives, like Selector and Labeler, which are the basic units of an interaction.
Figure 2. Portion of a Building Management Tasks and Interaction Model
Figure 2 presents a Task and Interaction Model for several tasks required in the building management applications that support building technicians, operators, and managers of buildings. The compositionality of each object in Figure 2 is indicated by lines between objects, with filled circles at the end of the line nearest the container object. For example, a Global Task (marked with a 1 on Figure 2) is composed of two subtasks, ActionSelect and ObjectSelect.
The TIM is a self-composing productive system. When an instance of a TIM object is created as an element in an active I-Space for a particular user and a particular set of domain objects, that instance is responsible for adapting to the current situation and producing its own parts. Each object fine-tunes its sub-components for the specific context, based on appropriateness to the situation and the user.
Using this process of self-composition, an entire user interface (or only the parts that need to be changed) can be created or modified in real time when needed. It is only necessary to tell a TIM application object that a building environmental control operator user interface is required, and all the frames, navigation hierarchies, graphics and buttons are created automatically. The user interface that is presented is specialized dynamically to suit the user, the objects in the real world, the interaction devices, and the required application tasks, and it dynamically responds to user input by redesigning specific parts of itself.
Figure 3. Generated Technical Task User Interface
Figure 3 shows a single task for a technical user. As shown in Figure 2 (marked with a 4), the Technical task is made up of two subtypes of Report (Specification and Description) and one definition subtask, called Maintainer. When the Technical task is presented to the user, these subtasks are shown in the bottom of a standard frame, arranged left to right. In addition, the Technical task inherits the Action Selection and Object Selection subtasks of the ViewObject Task that it specializes, which are ultimately presented as drop down menus across the top of the frame. Each of these five subtasks is marked with an arrow originating from the UISubTask box in Figure 3.
To illustrate how a subtask composes itself, consider the Maintainer subtask (bottom right of Figure 3). As shown in Figure 2, a Maintainer subtask (marked 5) is a specialization of the abstract Definition subtask (marked 6 in Figure 2). Each subtask selects the objects to include by matching the roles of the subtask and the roles of the data associated with the object of interest (in this case, the Freeze Stat). The Maintainer subtask looks for data associated with the Freeze Stat that has a maintenance role in the domain model. For each one that it finds, the Maintainer subtask creates one dynamic element. The elements in the Maintenance subtask are indicated by arrows from the UIElement box in the compositionality diagram at the top right of Figure 2.
The group of elements that appear in the Maintainer subtask are presented in editable form, since the user’s role includes maintenance. Thus, the Maintainer subtask is a subtype of the Definition subtask, which includes all tasks that involve editing or changing by a user. A Definition subtask is composed of dynamic elements, representing the individual information items as editable interactors. The class in the TIM that represents dynamic elements, DynamicInfo, is an abstract type, and cannot be used to create an object to represent an interaction component.
Definition subtasks can only be pre-defined abstractly, since it is impossible to know in advance what kinds of information must be expressed by each of the individual interaction components. An object like a Freeze Stat, for example, is a state switch, and so needs an On/Off switch to control its value, but a Thermostat, which can be set to a value from a constrained continuous range, needs a different type of interaction element as its control. In the same way, the interaction required to set a state switch to tell the system whether to suppress the Freeze state alarm is different from that required to enter the text value of the alarm message.
Since DynamicInfo is abstract, it must specialize itself whenever it appears in a Definition SubTask to match the properties of the data or object it represents, by choosing a matching specialization. As shown by arrows in Figure 2, DynamicInfo can specialize itself into other abstract elements representing sets or individual items, which can then be further specialized into instantiable elements appropriate for representing an individual piece of data. Each abstract element contains information about its specializations, and matches the data type constraints of the data it is trying to represent with the affordances of its specializations. For example, the Suppress Alarm data object associated with a Freeze Stat is represented by a State Data object in the domain model. All State Data objects have the data role of [+set] and [+state]. DynamicInfo matches [+set] to the OfSet specialization. OfSet, in turn, matches the [+state] constraint of the data object to the StateItem element, which is then instantiated to represent the interaction element for the Suppress Alarm data.
Conclusions and Future Research
We have discussed the Task and Interaction Model that DIGBE uses to automatically design interactions that dynamically adapt to the user and to the task and data environments. DIG demonstrates that interaction design knowledge can be modeled as an automated real time constraint-driven and affordance-driven process. It also demonstrates that this process can be separated from the processes necessary to present interactions on user interface devices and from those necessary to internally represent information about domain objects.
Additional areas of research are suggested by the current work. An extension of the current architecture to allow the incorporation of independently developed models of tasks, users, interaction design processes, and domain would greatly increase the utility of the system. Integration of a modularized, compositional DIG approach with efforts whose primary emphasis are the development of robust models would demonstrate both the real-world application of the models and the scalability of the DIG approach.
One promising example is the emerging work on task characterization [21]. This research has been successfully applied to automated visual discourse synthesis, using a model of presentation intents that is matched to a model of the visual tasks that achieve them. DIGBE attempts to include this sort of information through its dynamic specialization mechanisms, but the information is embedded in the interaction objects and not fully constrained by the roles in the domain that correspond to the intent structure in [21]. DIGBE is currently very good at specializing itself to different users and data roles, but the interaction design is still intimately woven into the task design. Further separation of tasks and interactions, similar to the separation of intents and tasks in visual discourse synthesis, would create a more modular DIG system and would improve the applicability of this type of approach to less well-understood task structures.
Automated interface design could become an economically beneficial alternative to static user interface design and user configuration in complex control domains. Our particular approach appears to be widely applicable to both legacy and future systems in a number of information intensive and variably configured applications, due to its reactivity to unique configurations and its separation between interaction and presentation. DIGBE demonstrates that, although full automatic synthesis of any interface at any time may not yet be feasible, it is both possible and useful to automatically generate interfaces to complex systems, particularly those with well understood task domains.
References
[1] Penner, R. Developing the process control interface. Engineering for Human computer Interaction. Amsterdam, Elsevier Scientific Press, 317-332, 1992.
[2] Penner, R., Nelson, K., and Soken, N. Facilitating human interactions in mixed-initiative systems. Proceedings of Systems, Man, and Cybernetics, 714-719, 1997.
[3] Penner, R. and Steinmetz, E. DIGBE: Adaptive user interface automation. Adaptive User Interfaces: Papers from the 2000 AAAI Spring Symposium (Technical Report SS-00-01),Menlo Park, California, AAAI Press, 98-101, 2000.
[4] Szekely, P. Retrospective and challenges for model-based interface development. In VanderDonckt, J., ed. Computer-Aided Design of User Interfaces, Presses Uniersitaires de Namur, 1996.
[5] Byrne, .M., Wood, S., Sukaviriya, P., Foley, J., Kieras, D. Automating user interface evaluation. Proceedings of the 1994 conference on Human Factors in Computing Systems (CHI94), 1994.
[6] Roth, S. and Mattis, J. Data characterization for intelligent graphics presentation. Proceedings of the Conference on Human Factors in Computing Systems (CHI ’90), 1990,. p. 193-200.
[7] Casner, S. A task-analytic approach to the automated design of graphic presentations. Transactions on Graphics, 10(2), 111-151, 1991.
[8] Seligman, D. and Feiner, S. Automated generation of intent-based 3D illustrations. Computer Graphics. 25(4), 123-132, 1991.
[9] Zhou, M., and Feiner, S. Data characterization for automatically visualizing hetrogeneous information. In Proceedings of IEEE Information Visualization ’96, 13-20, 1996.
[10] Zhou, M., and Feiner, S. Top-down hierarchical planning of coherent visual discourse. Proceedings of the 1997 International Conference on Intelligent User Interfaces, 129-136, 1997.
[11] Foley, J., Gibbs, C., Kim, W., and Kovacevic, S. Knowledge-based user interface management system. Proceedings of the 1988 Conference on Human Factors in Computer Systems (CHI ’88), 67-72, 1988.
[12] Wiecha, C., Bennett, W., Boies, S., Gould, J., and Greene, S. ITS: A tool for rapidly developing interactive applications. Transactions on Information Systems, 8(3), 204-236, 1989.
[13] Szekely, P., Luo, P., and Neches, R. Beyond interface builders: Model-based interface tools. Proceedings of INTERCHI ’93, 383-390, 1993.
[14] Puerta, A., Eriksson, H., Gennari, J., and Musen, M. Model-based automated generation of user interfaces. Proceedings of the National Conference on Artificial Intelligence, 471-477, 1994.
[15] Vanderdonckt, J. Reflections on automatic generation of a user interface for highly interactive business-oriented applications. In Maybury, M. and Wahlster, W., eds. Intelligent User Interfaces. San Francisco: Morgan Kaufmann Publishers, 1998
Monafred, R., Hodgson, A., Bowen, B., and West, A. Implementing a model-based generic user interface for computer integrated manufacturing systems. Proceedings of the Institution for Mechanical Engineering, Vol. l 212, Part B, 501-516, 1998.
[17] Penner, R. and Soken, N. Consistent Honeywell Interface: Tools for developers. Scientific Honeyweller, 106-109, 1993.
[18] Penner, R. Multimedia interfaces for process control. Proceedings of the Energy-Sources Technology Conference, ASME petroleum Division, 375-383, 1994.
[19] Galitz, W. The Essential Guide to User Interface Design. New York: Wiley, 1997.
[20] Mandel, T. The Elements of User Interface Design. New York: Wiley, 1997.
[21] Zhou, M. and Feiner, S. Visual task characterization for automated visual discourse synthesis. Proceedings of CHI98: Human Factors in Computing Systems, 392-399, 1998