UIST 2022

BO as Assistant: Using Bayesian Optimization for Asynchronously Generating Design Suggestions

National Institute of Advanced Industrial Science and Technology (AIST)

Background: Human-in-the-Loop BO

Bayesian optimization (BO) is a powerful technique characterized by its intelligent sampling. Previous work has used BO in human (designer)-in-the-loop frameworks for efficiently solving design problems. However, such frameworks are not flexible for designers, which may reduce agency and creativity.

Our Framework: "BO as Assistant"

We use BO as an assistant (rather than in human-in-the-loop), where designers can take the initiative in the design process while also benefiting from BO's intelligent sampling strategy. The system generates design suggestions using BO just by observing slider manipulation (without requiring explicit input).


The proposed framework is domain-agnostic. We demonstrate its generality by applying it to diverse design domains in computer graphics.

Main Video [4:56]

Preview Video [0:30]


Many design tasks involve parameter adjustment, and designers often struggle to find desirable parameter value combinations by manipulating sliders back and forth. For such a multi-dimensional search problem, Bayesian optimization (BO) is a promising technique because of its intelligent sampling strategy; in each iteration, BO samples the most effective points considering both exploration (i.e., prioritizing unexplored regions) and exploitation (i.e., prioritizing promising regions), enabling efficient searches. However, existing BO-based design frameworks take the initiative in the design process and thus are not flexible enough for designers to freely explore the design space using their domain knowledge. In this paper, we propose a novel design framework, BO as Assistant, which enables designers to take the initiative in the design process while also benefiting from BO's sampling strategy. The designer can manipulate sliders as usual; the system monitors the slider manipulation to automatically estimate the design goal on the fly and then asynchronously provides unexplored-yet-promising suggestions using BO's sampling strategy. The designer can choose to use the suggestions at any time. This framework uses a novel technique to automatically extract the necessary information to run BO by observing slider manipulation without requesting additional inputs. Our framework is domain-agnostic, demonstrated by applying it to photo color enhancement, 3D shape design for personal fabrication, and procedural material design in computer graphics.

Figure: Concept of BO as Assistant, a framework for assisting designers in finding appropriate design parameter values by using Bayesian optimization (BO) techniques. The system (1) monitors the designer's slider manipulation, (2) automatically estimates the designer's design goal, and (3) asynchronously provides suggestions sampled using BO's strategy. The designer can choose to use suggestions or ignore them.


Yuki Koyama and Masataka Goto. 2022. BO as Assistant: Using Bayesian Optimization for Asynchronously Generating Design Suggestions. In Proceedings of the 35th Annual ACM Symposium on User Interface Software and Technology (UIST '22), pp.77:1–77:14.

DOI: 10.1145/3526113.3545664


Python Implementation Guide


Our Bayesian optimization implementation is reproducible by using an open-source library and following the instruction below.


Our framework uses preferential Bayesian optimization (PBO), a variant of BO that runs with relative preferential information as input. First, you need to install a Python library that implements PBO, sequential-line-search, to your Python environment. This library can be installed using pip, but before that, you need to install CMake and Eigen.


brew install cmake eigen

Ubuntu 18.04:

sudo apt install cmake libeigen3-dev

Once Cmake and Eigen are installed, then the library (v0.3 is the version used in the paper) can be installed by the pip command.

pip install git+https://github.com/yuki-koyama/sequential-line-search.git@v0.3

Import Libraries

In your code, first you need to import the relevant libraries.

import pySequentialLineSearch as pysls
import numpy as np

Instantiate Optimizer

PreferentialBayesianOptimizer is the main class. As written in the paper, we use Gaussian Process Upper Confidence Bound (GP-UBC). The GP-UBC hyperparameter value is 0.5. We do not use the maximum a posteriori (MAP) estimation of kernel hyperparameters.

TYPE = pysls.AcquisitionFuncType.GaussianProcessUpperConfidenceBound

optimizer = pysls.PreferentialBayesianOptimizer(
    num_options=NUM_SUGGESTIONS + 1)


Note that the value of the num_options argument should be the number of the suggestions plus one as above.

Generate Initial Suggestions

At the beginning of the design process, there is no data available. So, we generate initial suggestions just randomly.

suggestions = np.random.rand(NUM_SUGGESTIONS, NUM_PARAMS)

Generate Suggestions

This is the main part of the PBO process. First, you need to extract a new preferential data entry from slider observation. Then, feed the data to the optimizer to update the internal model, and ask the optimizer to generate new suggestions. Finally, you need to retrieve the suggestions (and provide them to the user).

# Extract preferential feedback data
preferred_option, other_options = extract_preference_data(...)

# Submit the preferential data and update the internal model
optimizer.submit_custom_feedback_data(preferred_option, other_options)

# Calculate new suggestions

# Retrieve the options sampled by Bayesian optimization
for i in range(NUM_SUGGESTIONS):
    suggestions[i] = optimizer.get_current_options()[i + 1]

Note that optimizer.get_current_options()[0] always stores the "current best" option and so can be just ignored in this work.

Related Projects

Photographic Lighting Design with Photographer-in-the-Loop Bayesian Optimization

Kenta Yamamoto, Yuki Koyama, and Yoichi Ochiai

UIST 2022

Sequential Gallery for Interactive Visual Design Optimization

Yuki Koyama, Issei Sato, and Masataka Goto

ACM Transactions on Graphics (SIGGRAPH 2020)

Sequential Line Search for Efficient Visual Design Optimization by Crowds

Yuki Koyama, Issei Sato, Daisuke Sakamoto, and Takeo Igarashi

ACM Transactions on Graphics (SIGGRAPH 2017)


Yuki Koyama

is a Senior Researcher at National Institute of Advanced Industrial Science and Technology (AIST), Japan. His main research area is Computer Graphics and Human-Computer Interaction, specializing in computational design techniques.

Masataka Goto

is a Prime Senior Researcher at National Institute of Advanced Industrial Science and Technology (AIST), Japan. In 2016, as the Research Director he began a 5-year research project (JST OngaACCEL Project) on music technologies.