Benchmarks with Kurobako

Here we show a few, not exclusive benchmarks of Optuna hyperparameter optimization. We expect this to help you understand which combination of sampling algorithms and pruning algorithms best suits your problem.

We use sile/kurobako (kurobako means "black box" in Japanese). Those who are familiar with Japanese, we would highly recommend https://search.ieice.org/bin/pdf_link.php?category=D&lang=J&year=2020&fname=j103-d_9_615&abst= as a guideline.

Benchmark

How to Run

Below is the example script to run the benchmark of optuna.samplers.TPESampler and optuna.pruners.NopPruner. The first argument is required if your kurobako installation path is not in $PATH. Note that the below script is for kurobako v0.1.

$ sh ./runner.sh <directory of kurobako command>

Click to toggle contents of shell script

#!/bin/bash

name=tpe_nop
MIN_RESOURCE=1
MIN_RESOURCE_HPO=1
MIN_RESOURCE_NAS=4
REDUCTION_FACTOR=3
N_WARMUP_STEPS=40
N_RUN=100
BUDGET=80
BUDGET_HPO=80
BUDGET_NAS=120
INPUT_DIR=data
OUTPUT_DIR=results
sampler_list=(tpe)
pruner_list=(nop)
problem_list=(hpobench nasbench)
K_PATH="$1"

mkdir -p "${OUTPUT_DIR}" 2>/dev/null
mkdir -p "${OUTPUT_DIR}"/studies/${name}
echo -n >| "${OUTPUT_DIR}"/studies/${name}/studies_${name}.json

for problem in "${problem_list[@]}"; do
  echo -n >| "${OUTPUT_DIR}"/solvers.json
  echo -n >| "${OUTPUT_DIR}"/problems.json

  if [ "${problem}" == hpobench ]; then
    datasets=(
      "${INPUT_DIR}"/fcnet_naval_propulsion_data.hdf5
      "${INPUT_DIR}"/fcnet_parkinsons_telemonitoring_data.hdf5
      "${INPUT_DIR}"/fcnet_protein_structure_data.hdf5
      "${INPUT_DIR}"/fcnet_slice_localization_data.hdf5
    )
    MIN_RESOURCE=${MIN_RESOURCE_HPO}
    BUDGET=${BUDGET_HPO}
  elif [ "${problem}" == nasbench ]; then
    datasets=(
      "${INPUT_DIR}"/nasbench_full.bin
      # "${INPUT_DIR}"/nasbench_only108.bin
    )
    MIN_RESOURCE=${MIN_RESOURCE_NAS}
    BUDGET=${BUDGET_NAS}
  elif [ "${problem}" == sigopt ]; then
    datasets=(
      rosenbrock-log
      six-hump-camel
    )
  fi

  if [ "${problem}" == hpobench ] || [ "${problem}" == nasbench ]; then
    for dataset in "${datasets[@]}"; do
      "${K_PATH}"kurobako problem "${problem}" "${dataset}" | tee -a "${OUTPUT_DIR}"/problems.json
    done
  elif [ "${problem}" == sigopt ]; then
    for dataset in "${datasets[@]}"; do
      "${K_PATH}"kurobako problem "${problem}" "${dataset}" | tee -a "${OUTPUT_DIR}"/problems.json
    done
  fi

  for sampler in "${sampler_list[@]}"; do
    for pruner in "${pruner_list[@]}"; do
      "${K_PATH}"kurobako solver --name "${sampler}"-"${pruner}" optuna \
        --loglevel debug \
        --sampler "${sampler}" \
        --skopt-base-estimator GP \
        --pruner "${pruner}" \
        --hyperband-min-resource ${MIN_RESOURCE} \
        --hyperband-reduction-factor ${REDUCTION_FACTOR} \
         --median-warmup-steps ${N_WARMUP_STEPS} \
    | tee -a "${OUTPUT_DIR}"/solvers.json
  done

  "${K_PATH}"kurobako studies \
    --solvers $(cat "${OUTPUT_DIR}"/solvers.json) \
    --problems $(cat "${OUTPUT_DIR}"/problems.json) \
    --repeats ${N_RUN} \
    --budget ${BUDGET} \
  >> "${OUTPUT_DIR}"/studies/${name}/studies_${name}.json
done
done

mkdir -p "${OUTPUT_DIR}"/results/${name}
cat "${OUTPUT_DIR}"/studies/${name}/studies_${name}.json | "${K_PATH}"kurobako run --parallelism 10 > "${OUTPUT_DIR}"/results/${name}/result_${name}.json

mkdir -p "${OUTPUT_DIR}"/reports/${name}
cat "${OUTPUT_DIR}"/results/${name}/result_${name}.json | "${K_PATH}"kurobako report > "${OUTPUT_DIR}"/reports/${name}/report_${name}.md

1. optuna.samplers.RandomSampler

• Optuna: v2.0.0
• kurobako: v0.1.14

2. optuna.samplers.TPESampler

• Optuna: v2.0.0
• kurobako: v0.1.14

3. optuna.samplers.CmaEsSampler

• Optuna: v2.1.0
• kurobako: v0.2.6

Note: In the following result, we set consider_pruned_trials=True.

Note: In the following result, we set consider_pruned_trials=False.