CHI 2021 Courses

# Introduction to Computational Design

Instructor: Yuki Koyama

## Quick Facts

• This is a 75-minute (1-unit) course.
• This is an introductory course on computational design, one of the hot topics in HCI and related research fields.
• Anyone with interest in computational design can attend this course. Novices to this topic are especially welcomed.
• The exact day and time will be announced later.

## Abstract

Computational design is one of the hot topics in HCI and related research fields, where various design problems are formulated using mathematical languages and solved by computational techniques. By this paradigm, researchers aim at establishing highly sophisticated or efficient design processes that otherwise cannot be achieved. Target domains include graphics, personal fabrication, user interface, etc. This course introduces fundamental concepts in computational design and provides an overview of the recent trend. It then goes into a more specific case where human assessment is necessary to evaluate the quality of design outcomes, which is often true in HCI scenarios. This course is recommended to HCI students and researchers who are new to this topic.

## Introduction

### What is Computational Design?

#### Definition

Computational techniques, such as optimization and machine learning, have recently played an increasingly important role in enhancing human-computer interaction (HCI). Computational design is one of the emerging hot topics in this context. Since there has seemed no established definition yet (Bi et al. 2017), we define it as follows and will use it in this course:

Computational design is a paradigm in which design problems are formulated mathematically and solved by computational techniques.

The formulated design problem takes the form of optimization in most cases (Bi et al. 2017), and the optimization is performed as the process of searching for the best design among some options either in an automatic way by running optimization algorithms, in a manual way with computationally enabled support, or in a hybrid way using computational systems with having a human in the loop (Koyama and Igarashi 2018). The primary goal of computational design research is to establish sophisticated or efficient design processes that otherwise cannot be achieved (Koyama and Igarashi 2018). This topic has also been actively studied in related fields, such as computer graphics. Application domains include visual design (Koyama et al. 2017; Koyama, Sato, and Goto 2020), personal fabrication (Umetani et al. 2014; Prévost et al. 2013), user interface (Oulasvirta et al. 2019), interactive device (Fujinawa et al. 2017; Bächer et al. 2016), robotics (Megaro et al. 2015), etc.

#### Design as Optimization

Typical optimization problems can be described using the mathematical language in the form of \begin{aligned}\mathbf{x}^{*} = \mathop{\mathrm{arg~max}}_{\mathbf{x} \in \mathcal{X}} f(\mathbf{x}). \label{eq:optimization}\end{aligned} The message of this equation is simple: the variable set $$\mathbf{x}^{*}$$ is the maximizer of the function $$f$$ within the candidate space $$\mathcal{X}$$. See the following figure for illustration. This equation involves the following concepts, each of which can be interpreted as a concept in design:

Search variable set $$\textbf{x}$$:

This concept is about design parameter set in the design context and determines which parameters will be computationally manipulated in the optimization process. It can consist of either a single parameter or multiple parameters. For example, it can include font size, font color, background color, etc., in web design scenarios. These parameters can be either discrete (e.g., font) or continuous (e.g., font size).

Search space $$\mathcal{X}$$:
This concept is about design space in the design context and represents the set of all the possible design choices. If the search variables are discrete, then the search space is a list of candidates (e.g., $$\mathcal{X} =$$ { "Times", "Helvetica" } if $$\mathbf{x}$$ consists of a font parameter). If the search variables are continuous, then the search space can also be continuous.

Objective function $$f$$:

This concept is about design goal and determines the criterion based on which the design artifact is evaluated. In other others, this function quantifies how good a given choice is.

Optimal solution $$\mathbf{x}^{*}$$:

This concept represents the variable set that provides the best possible design. Here, “best” means that the value of the objective function is maximized (or minimized, depending on the formulation). This is found as the outcome of the optimization process.

We can describe many design problems using these concepts. This course will review representative works from various design domains and explain how researchers have formulated actual design problems using these concepts.

Designing an appropriate objective function is the key to success and often requires an extensive understanding of the target design problem. It can be implemented using predictive models of human perception or behavior, which can be either rule-based or data-driven, physical simulation, direct query-by-query response from human evaluators, etc.

Once the target design problem is formulated as an optimization problem, we can solve it by applying existing optimization algorithms (e.g., gradient descent, Newton’s method, and simulated annealing) from existing libraries (e.g., SciPy (Virtanen et al. 2020)) in most cases. Note that researchers need to choose an appropriate optimization algorithm that is compatible with the target problem. This course will explain basic considerations for the choice, but for those interested in further details, we suggest referring to the book (Nocedal and Wright 2006).

### Human in the Loop

Objective functions often involve subjective preferential evaluation. For example, the goal of photo color enhancement is typically to find the most “subjectively pleasing” photo enhancement by adjusting parameters such as brightness, contrast, and saturation (Koyama, Sakamoto, and Igarashi 2016). In such cases, it is often difficult to implement appropriate objective functions since accurately predicting human preference is a challenging task.

Human-in-the-loop optimization is an effective approach to handling human preference in computational design systems. In this approach, the computational design system iteratively asks human evaluators (e.g., the user (Koyama, Sato, and Goto 2020) or crowd workers (Koyama et al. 2017)) to perform some microtasks while running the optimization process. That is, the system considers the human evaluators as a processing module that plays the role of the objective function, in the spirit of human computation (Quinn and Bederson 2011). Another effective approach is to use preference learning techniques (Koyama, Sakamoto, and Igarashi 2014, 2016) for approximating the preferential objective function and then let users manually find the optimal solution with the help of this approximated objective function.

When gathering subjective preferential feedback from human evaluators, we need to care about how to design queries. In general, researchers consider relative assessment (e.g., let evaluators choose the best design from multiple options) to be more suitable than absolute assessment (e.g., let evaluators provide a score value for a specific design) (Tsukida and Gupta 2011; Koyama and Igarashi 2018). This course will discuss how this consideration has been incorporated into computational design frameworks by reviewing several representative works.

### Examples

The following figure shows an example of computational design for personal fabrication (Umetani et al. 2014), in which a system for designing free-form model airplanes is presented. To ensure that the airplane can fly well without letting users perform time-consuming fabricate-and-test iterations, the system automatically adjusts design parameters (e.g., wing positions) by solving an optimization problem, where the “flyability” (i.e., how long and stably the airplane can fly) is considered the objective function to maximize.

The following figure shows an example of human-in-the-loop computational design for parametric visual design (Koyama, Sato, and Goto 2020), in which the presented human-in-the-loop optimization method is applied to photo color enhancement. In this design problem, the user needs to adjust design parameters (e.g., brightness, contrast, and saturation) such that the target photograph looks subjectively best. The system finds the optimal parameter set by iteratively asking the user to perform a simple assessment task (clicking the best option among some options in this case).

• Bächer, Moritz, Benjamin Hepp, Fabrizio Pece, Paul G. Kry, Bernd Bickel, Bernhard Thomaszewski, and Otmar Hilliges. 2016. DefSense: Computational Design of Customized Deformable Input Devices.” In Proc. CHI ’16, 3806–16. [DOI]
• Bi, Xiaojun, Otmar Hilliges, Takeo Igarashi, and Antti Oulasvirta. 2017. Computational Interactivity (Dagstuhl Seminar 17232).” Edited by Xiaojun Bi, Otmar Hilliges, Takeo Igarashi, and Antti Oulasvirta. Dagstuhl Reports 7 (6): 48–67. [DOI]
• Fujinawa, Eisuke, Shigeo Yoshida, Yuki Koyama, Takuji Narumi, Tomohiro Tanikawa, and Michitaka Hirose. 2017. “Computational Design of Hand-Held VR Controllers Using Haptic Shape Illusion.” In Proc. VRST ’17, 28:1–10. [DOI]
• Koyama, Yuki, and Takeo Igarashi. 2018. “Computational Design with Crowds.” In Computational Interaction, edited by Antti Oulasvirta, Per Ola Kristensson, Xiaojun Bi, and Andrew Howes, 153–84. Oxford University Press. [DOI]
• Koyama, Yuki, Daisuke Sakamoto, and Takeo Igarashi. 2014. “Crowd-Powered Parameter Analysis for Visual Design Exploration.” In Proc. UIST ’14, 65–74. [DOI]
• Koyama, Yuki, Daisuke Sakamoto, and Takeo Igarashi. 2016. SelPh: Progressive Learning and Support of Manual Photo Color Enhancement.” In Proc. CHI ’16, 2520–32. [DOI]
• Koyama, Yuki, Issei Sato, and Masataka Goto. 2020. “Sequential Gallery for Interactive Visual Design Optimization.” ACM Trans. Graph. 39 (4): 88:1–12. [DOI]
• Koyama, Yuki, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi. 2017. “Sequential Line Search for Efficient Visual Design Optimization by Crowds.” ACM Trans. Graph. 36 (4): 48:1–11. [DOI]
• Megaro, Vittorio, Bernhard Thomaszewski, Maurizio Nitti, Otmar Hilliges, Markus Gross, and Stelian Coros. 2015. “Interactive Design of 3d-Printable Robotic Creatures.” ACM Trans. Graph. 34 (6): 216:1–9. [DOI]
• Nocedal, Jorge, and Stephen J. Wright. 2006. Numerical Optimization. 2nd ed. Springer Science+Business Media. [DOI]
• Oulasvirta, Antti, Niraj Ramesh Dayama, Morteza Shiripour, Maximilian John, and Andreas Karrenbauer. 2019. “Combinatorial Optimization of Graphical User Interface Designs.” Proc. IEEE 108 (3): 434–64. [DOI]
• Prévost, Romain, Emily Whiting, Sylvain Lefebvre, and Olga Sorkine-Hornung. 2013. “Make It Stand: Balancing Shapes for 3d Fabrication.” ACM Trans. Graph. 32 (4): 81:1–10. [DOI]
• Quinn, Alexander J., and Benjamin B. Bederson. 2011. “Human Computation: A Survey and Taxonomy of a Growing Field.” In Proc. CHI ’11, 1403–12. [DOI]
• Tsukida, Kristi, and Maya R. Gupta. 2011. “How to Analyze Paired Comparison Data.” UWEETR-2011-0004. University of Washington, Department of Electrical Engineering. https://vannevar.ece.uw.edu/techsite/papers/refer/UWEETR-2011-0004.html.
• Umetani, Nobuyuki, Yuki Koyama, Ryan Schmidt, and Takeo Igarashi. 2014. “Pteromys: Interactive Design and Optimization of Free-Formed Free-Flight Model Airplanes.” ACM Trans. Graph. 33 (4): 65:1–10. [DOI]
• Virtanen, Pauli, Ralf Gommers, Travis E. Oliphant, Matt Haberland, Tyler Reddy, David Cournapeau, Evgeni Burovski, et al. 2020. SciPy 1.0: Fundamental Algorithms for Scientific Computing in Python.” Nature Methods 17: 261–72. [DOI]

## Benefits

This course aims to let students and researchers who are novices to computational design become ready to start research on this topic. Specifically, this course tries to offer the following benefits to the audience:

• Understanding of fundamental concepts in computational design and an overview of this topic.
• Understanding of how researchers have formulated design problems as optimization in various application domains.
• Understanding of common considerations when human evaluation is involved in optimization problems.

## Intended Audiences

Anyone with interest in computational design can attend this course. Students, researchers, and practitioners who are novices to this topic are especially welcomed.

## Prerequisites

This course is self-contained, and no particular prerequisite knowledge is required. This course will involve some basic concepts of mathematics and coding, but we will explain the necessary concepts during the course.

## Course Content

This 75-minute course consists of three parts. In the first two parts, we will introduce many representative works from various application domains. This is not intended as a comprehensive survey but as material for a better understanding of computational design, and so we will focus on explaining why and how each of the works can be considered computational design. The last part summarizes the course and mentions future challenges.

• Part 0: Course Introduction (5 minutes)
• Part 1: What is Computational Design? (35 minutes)
• Part 2: Human in the Loop (25 minutes)
• Part 3: Summary (10 minutes)

## Instructor Background

Yuki Koyama is a Researcher at National Institute of Advanced Industrial Science and Technology (AIST), Japan. He has been working on computational design for years in HCI and graphics domains and published papers on this topic at important venues such as CHI, UIST, SIGGRAPH, and SIGGRAPH Asia. His recent interest is to apply computational techniques for formulating and supporting design processes that involve preferential assessment by human; for example, he has worked on supporting parametric visual designs using crowdsourcing, machine learning, and Bayesian methods [UIST 2014] [CHI 2016] [SIGGRAPH 2017] [SIGGRAPH 2020]. He is also interested in solving computational design problems in the personal fabrication domain [SIGGRAPH 2014] [SIGGRAPH Asia 2015] [VRST 2017] [CHI 2021]. He received his Ph.D. from the University of Tokyo in 2017.

Personal webpage: https://koyama.xyz/