Computational design is one of the hot topics in HCI and related research fields, where various design problems are formulated using mathematical languages and solved by computational techniques. By this paradigm, researchers aim at establishing highly sophisticated or efficient design processes that otherwise cannot be achieved. Target domains include graphics, personal fabrication, user interface, etc. This course introduces fundamental concepts in computational design and provides an overview of the recent trend. It then goes into a more specific case where human assessment is necessary to evaluate the quality of design outcomes, which is often true in HCI scenarios. This course is recommended to HCI students and researchers who are new to this topic.
Computational techniques, such as optimization and machine learning, have recently played an increasingly important role in enhancing human-computer interaction (HCI). Computational design is one of the emerging hot topics in this context. Since there has seemed no established definition yet (Bi et al. 2017), we define it as follows and will use it in this course:
Computational design is a paradigm in which design problems are formulated mathematically and solved by computational techniques.
The formulated design problem takes the form of optimization in most cases (Bi et al. 2017), and the optimization is performed as the process of searching for the best design among some options either in an automatic way by running optimization algorithms, in a manual way with computationally enabled support, or in a hybrid way using computational systems with having a human in the loop (Koyama and Igarashi 2018). The primary goal of computational design research is to establish sophisticated or efficient design processes that otherwise cannot be achieved (Koyama and Igarashi 2018). This topic has also been actively studied in related fields, such as computer graphics. Application domains include visual design (Koyama et al. 2017; Koyama, Sato, and Goto 2020), personal fabrication (Umetani et al. 2014; Prévost et al. 2013), user interface (Oulasvirta et al. 2019), interactive device (Fujinawa et al. 2017; Bächer et al. 2016), robotics (Megaro et al. 2015), etc.
Typical optimization problems can be described using the mathematical language in the form of \[\begin{aligned}\mathbf{x}^{*} = \mathop{\mathrm{arg~max}}_{\mathbf{x} \in \mathcal{X}} f(\mathbf{x}). \label{eq:optimization}\end{aligned}\] The message of this equation is simple: the variable set \(\mathbf{x}^{*}\) is the maximizer of the function \(f\) within the candidate space \(\mathcal{X}\). See the following figure for illustration. This equation involves the following concepts, each of which can be interpreted as a concept in design:
This concept is about design parameter set in the design context and determines which parameters will be computationally manipulated in the optimization process. It can consist of either a single parameter or multiple parameters. For example, it can include font size, font color, background color, etc., in web design scenarios. These parameters can be either discrete (e.g., font) or continuous (e.g., font size).
This concept is about design goal and determines the criterion based on which the design artifact is evaluated. In other others, this function quantifies how good a given choice is.
This concept represents the variable set that provides the best possible design. Here, “best” means that the value of the objective function is maximized (or minimized, depending on the formulation). This is found as the outcome of the optimization process.
We can describe many design problems using these concepts. This course will review representative works from various design domains and explain how researchers have formulated actual design problems using these concepts.
Designing an appropriate objective function is the key to success and often requires an extensive understanding of the target design problem. It can be implemented using predictive models of human perception or behavior, which can be either rule-based or data-driven, physical simulation, direct query-by-query response from human evaluators, etc.
Once the target design problem is formulated as an optimization problem, we can solve it by applying existing optimization algorithms (e.g., gradient descent, Newton’s method, and simulated annealing) from existing libraries (e.g., SciPy (Virtanen et al. 2020)) in most cases. Note that researchers need to choose an appropriate optimization algorithm that is compatible with the target problem. This course will explain basic considerations for the choice, but for those interested in further details, we suggest referring to the book (Nocedal and Wright 2006).
Objective functions often involve subjective preferential evaluation. For example, the goal of photo color enhancement is typically to find the most “subjectively pleasing” photo enhancement by adjusting parameters such as brightness, contrast, and saturation (Koyama, Sakamoto, and Igarashi 2016). In such cases, it is often difficult to implement appropriate objective functions since accurately predicting human preference is a challenging task.
Human-in-the-loop optimization is an effective approach to handling human preference in computational design systems. In this approach, the computational design system iteratively asks human evaluators (e.g., the user (Koyama, Sato, and Goto 2020) or crowd workers (Koyama et al. 2017)) to perform some microtasks while running the optimization process. That is, the system considers the human evaluators as a processing module that plays the role of the objective function, in the spirit of human computation (Quinn and Bederson 2011). Another effective approach is to use preference learning techniques (Koyama, Sakamoto, and Igarashi 2014, 2016) for approximating the preferential objective function and then let users manually find the optimal solution with the help of this approximated objective function.
When gathering subjective preferential feedback from human evaluators, we need to care about how to design queries. In general, researchers consider relative assessment (e.g., let evaluators choose the best design from multiple options) to be more suitable than absolute assessment (e.g., let evaluators provide a score value for a specific design) (Tsukida and Gupta 2011; Koyama and Igarashi 2018). This course will discuss how this consideration has been incorporated into computational design frameworks by reviewing several representative works.
The following figure shows an example of computational design for personal fabrication (Umetani et al. 2014), in which a system for designing free-form model airplanes is presented. To ensure that the airplane can fly well without letting users perform time-consuming fabricate-and-test iterations, the system automatically adjusts design parameters (e.g., wing positions) by solving an optimization problem, where the “flyability” (i.e., how long and stably the airplane can fly) is considered the objective function to maximize.
The following figure shows an example of human-in-the-loop computational design for parametric visual design (Koyama, Sato, and Goto 2020), in which the presented human-in-the-loop optimization method is applied to photo color enhancement. In this design problem, the user needs to adjust design parameters (e.g., brightness, contrast, and saturation) such that the target photograph looks subjectively best. The system finds the optimal parameter set by iteratively asking the user to perform a simple assessment task (clicking the best option among some options in this case).
This course aims to let students and researchers who are novices to computational design become ready to start research on this topic. Specifically, this course tries to offer the following benefits to the audience:
Anyone with interest in computational design can attend this course. Students, researchers, and practitioners who are novices to this topic are especially welcomed.
This course is self-contained, and no particular prerequisite knowledge is required. This course will involve some basic concepts of mathematics and coding, but we will explain the necessary concepts during the course.
This 75-minute course consists of three parts. In the first two parts, we will introduce many representative works from various application domains. This is not intended as a comprehensive survey but as material for a better understanding of computational design, and so we will focus on explaining why and how each of the works can be considered computational design. The last part summarizes the course and mentions future challenges.
Yuki Koyama is a Researcher at National Institute of Advanced Industrial Science and Technology (AIST), Japan. He has been working on computational design for years in HCI and graphics domains and published papers on this topic at important venues such as CHI, UIST, SIGGRAPH, and SIGGRAPH Asia. His recent interest is to apply computational techniques for formulating and supporting design processes that involve preferential assessment by human; for example, he has worked on supporting parametric visual designs using crowdsourcing, machine learning, and Bayesian methods [UIST 2014] [CHI 2016] [SIGGRAPH 2017] [SIGGRAPH 2020]. He is also interested in solving computational design problems in the personal fabrication domain [SIGGRAPH 2014] [SIGGRAPH Asia 2015] [VRST 2017] [CHI 2021]. He received his Ph.D. from the University of Tokyo in 2017.
Personal webpage: https://koyama.xyz/
