Skip to content

Neo LS-SVM is a modern Least-Squares Support Vector Machine implementation

License

Notifications You must be signed in to change notification settings

lsorber/neo-ls-svm

Repository files navigation

Open in Dev Containers Open in GitHub Codespaces

Neo LS-SVM

Neo LS-SVM is a modern Least-Squares Support Vector Machine implementation in Python that offers several benefits over sklearn's classic sklearn.svm.SVC classifier and sklearn.svm.SVR regressor:

  1. ⚑ Linear complexity in the number of training examples with Orthogonal Random Features.
  2. πŸš€ Hyperparameter free: zero-cost optimization of the regularisation parameter Ξ³ and kernel parameter Οƒ.
  3. πŸ”οΈ Adds a new tertiary objective that minimizes the complexity of the prediction surface.
  4. 🎁 Returns the leave-one-out residuals and error for free after fitting.
  5. πŸŒ€ Learns an affine transformation of the feature matrix to optimally separate the target's bins.
  6. πŸͺž Can solve the LS-SVM both in the primal and dual space.
  7. 🌑️ Isotonically calibrated predict_proba.
  8. βœ… Conformally calibrated predict_quantiles and predict_interval.
  9. πŸ”” Bayesian estimation of the predictive standard deviation with predict_std.
  10. 🐼 Pandas DataFrame output when the input is a pandas DataFrame.

Using

Installing

First, install this package with:

pip install neo-ls-svm

Classification and regression

Then, you can import neo_ls_svm.NeoLSSVM as an sklearn-compatible binary classifier and regressor. Example usage:

from neo_ls_svm import NeoLSSVM
from pandas import get_dummies
from sklearn.datasets import fetch_openml
from sklearn.model_selection import train_test_split

# Binary classification example:
X, y = fetch_openml("churn", version=3, return_X_y=True, as_frame=True, parser="auto")
X_train, X_test, y_train, y_test = train_test_split(get_dummies(X), y, test_size=0.15, random_state=42)
model = NeoLSSVM().fit(X_train, y_train)
model.score(X_test, y_test)  # 93.1% (compared to sklearn.svm.SVC's 89.6%)

# Regression example:
X, y = fetch_openml("ames_housing", version=1, return_X_y=True, as_frame=True, parser="auto")
X_train, X_test, y_train, y_test = train_test_split(get_dummies(X), y, test_size=0.15, random_state=42)
model = NeoLSSVM().fit(X_train, y_train)
model.score(X_test, y_test)  # 82.4% (compared to sklearn.svm.SVR's -11.8%)

Predicting quantiles

Neo LS-SVM implements conformal prediction with a Bayesian nonconformity estimate to compute quantiles and prediction intervals for both classification and regression. Example usage:

# Predict the house prices and their quantiles.
Ε·_test = model.predict(X_test)
Ε·_test_quantiles = model.predict_quantiles(X_test, quantiles=(0.025, 0.05, 0.1, 0.9, 0.95, 0.975))

When the input data is a pandas DataFrame, the output is also a pandas DataFrame. For example, printing the head of Ε·_test_quantiles yields:

house_id 0.025 0.05 0.1 0.9 0.95 0.975
1357 114283.0 124767.6 133314.0 203162.0 220407.5 245655.3
2367 85518.3 91787.2 93709.8 107464.3 108472.6 114482.3
2822 147165.9 157462.8 167193.1 243646.5 263324.4 291963.3
2126 81788.7 88738.1 91367.4 111944.9 114800.7 122874.5
1544 94507.1 108288.2 120184.3 222630.5 248668.2 283703.4

Let's visualize the predicted quantiles on the test set:

Expand to see the code that generated the graph above
import matplotlib.pyplot as plt
import matplotlib.ticker as ticker

%config InlineBackend.figure_format = "retina"
plt.rcParams["font.size"] = 8
idx = (-Ε·_test.sample(50, random_state=42)).sort_values().index
y_ticks = list(range(1, len(idx) + 1))
plt.figure(figsize=(4, 5))
for j in range(3):
    end = Ε·_test_quantiles.shape[1] - 1 - j
    coverage = round(100 * (Ε·_test_quantiles.columns[end] - Ε·_test_quantiles.columns[j]))
    plt.barh(
        y_ticks,
        Ε·_test_quantiles.loc[idx].iloc[:, end] - Ε·_test_quantiles.loc[idx].iloc[:, j],
        left=Ε·_test_quantiles.loc[idx].iloc[:, j],
        label=f"{coverage}% Prediction interval",
        color=["#b3d9ff", "#86bfff", "#4da6ff"][j],
    )
plt.plot(y_test.loc[idx], y_ticks, "s", markersize=3, markerfacecolor="none", markeredgecolor="#e74c3c", label="Actual value")
plt.plot(Ε·_test.loc[idx], y_ticks, "s", color="blue", markersize=0.6, label="Predicted value")
plt.xlabel("House price")
plt.ylabel("Test house index")
plt.xlim(0, 500e3)
plt.yticks(y_ticks, y_ticks)
plt.tick_params(axis="y", labelsize=6)
plt.grid(axis="x", color="lightsteelblue", linestyle=":", linewidth=0.5)
plt.gca().xaxis.set_major_formatter(ticker.StrMethodFormatter("${x:,.0f}"))
plt.gca().spines["top"].set_visible(False)
plt.gca().spines["right"].set_visible(False)
plt.legend()
plt.tight_layout()
plt.show()

Predicting intervals

In addition to quantile prediction, you can use predict_interval to predict conformally calibrated prediction intervals. Compared to quantiles, these focus on reliable coverage over quantile accuracy. Example usage:

# Compute prediction intervals for the houses in the test set.
Ε·_test_interval = model.predict_interval(X_test, coverage=0.95)

# Measure the coverage of the prediction intervals on the test set
coverage = ((Ε·_test_interval.iloc[:, 0] <= y_test) & (y_test <= Ε·_test_interval.iloc[:, 1])).mean()
print(coverage)  # 94.3%

When the input data is a pandas DataFrame, the output is also a pandas DataFrame. For example, printing the head of Ε·_test_interval yields:

house_id 0.025 0.975
1357 114283.0 245849.2
2367 85518.3 114411.4
2822 147165.9 292179.2
2126 81788.7 122838.1
1544 94507.1 284062.6

Benchmarks

We select all binary classification and regression datasets below 1M entries from the AutoML Benchmark. Each dataset is split into 85% for training and 15% for testing. We apply skrub.TableVectorizer as a preprocessing step for neo_ls_svm.NeoLSSVM and sklearn.svm.SVC,SVR to vectorize the pandas DataFrame training data into a NumPy array. Models are fitted only once on each dataset, with their default settings and no hyperparameter tuning.

Binary classification

ROC-AUC on 15% test set:

dataset LGBMClassifier NeoLSSVM SVC
ada πŸ₯ˆ 90.9% (0.1s) πŸ₯‡ 90.9% (1.9s) 83.1% (4.5s)
adult πŸ₯‡ 93.0% (0.5s) πŸ₯ˆ 89.0% (15.7s) /
amazon_employee_access πŸ₯‡ 85.6% (0.5s) πŸ₯ˆ 64.5% (9.0s) /
arcene πŸ₯ˆ 78.0% (0.6s) 70.0% (6.3s) πŸ₯‡ 82.0% (4.0s)
australian πŸ₯‡ 88.3% (0.2s) 79.9% (1.7s) πŸ₯ˆ 81.9% (0.1s)
bank-marketing πŸ₯‡ 93.5% (0.5s) πŸ₯ˆ 91.0% (11.8s) /
blood-transfusion-service-center 62.0% (0.3s) πŸ₯‡ 71.0% (2.2s) πŸ₯ˆ 69.7% (0.1s)
churn πŸ₯‡ 91.7% (0.6s) πŸ₯ˆ 81.0% (2.1s) 70.6% (2.9s)
click_prediction_small πŸ₯‡ 67.7% (0.5s) πŸ₯ˆ 66.6% (10.9s) /
jasmine πŸ₯‡ 86.1% (0.3s) 79.5% (1.9s) πŸ₯ˆ 85.3% (7.4s)
kc1 πŸ₯‡ 78.9% (0.3s) πŸ₯ˆ 76.6% (1.4s) 45.7% (0.6s)
kr-vs-kp πŸ₯‡ 100.0% (0.6s) 99.2% (1.6s) πŸ₯ˆ 99.4% (2.3s)
madeline πŸ₯‡ 93.1% (0.5s) 65.6% (1.9s) πŸ₯ˆ 82.5% (19.8s)
ozone-level-8hr πŸ₯ˆ 91.2% (0.4s) πŸ₯‡ 91.6% (1.7s) 72.9% (0.6s)
pc4 πŸ₯‡ 95.3% (0.3s) πŸ₯ˆ 90.9% (1.5s) 25.7% (0.3s)
phishingwebsites πŸ₯‡ 99.5% (0.5s) πŸ₯ˆ 98.9% (3.6s) 98.7% (10.0s)
phoneme πŸ₯‡ 95.6% (0.3s) πŸ₯ˆ 93.5% (2.1s) 91.2% (2.0s)
qsar-biodeg πŸ₯‡ 92.7% (0.4s) πŸ₯ˆ 91.1% (5.2s) 86.8% (0.3s)
satellite πŸ₯ˆ 98.7% (0.2s) πŸ₯‡ 99.5% (1.9s) 98.5% (0.4s)
sylvine πŸ₯‡ 98.5% (0.2s) πŸ₯ˆ 97.1% (2.0s) 96.5% (3.8s)
wilt πŸ₯ˆ 99.5% (0.2s) πŸ₯‡ 99.8% (1.8s) 98.9% (0.5s)
Regression

RΒ² on 15% test set:

dataset LGBMRegressor NeoLSSVM SVR
abalone πŸ₯ˆ 56.2% (0.1s) πŸ₯‡ 59.5% (2.5s) 51.3% (0.7s)
boston πŸ₯‡ 91.7% (0.2s) πŸ₯ˆ 89.6% (1.1s) 35.1% (0.0s)
brazilian_houses πŸ₯ˆ 55.9% (0.3s) πŸ₯‡ 88.4% (3.7s) 5.4% (7.0s)
colleges πŸ₯‡ 58.5% (0.4s) πŸ₯ˆ 42.2% (6.6s) 40.2% (15.1s)
diamonds πŸ₯‡ 98.2% (0.3s) πŸ₯ˆ 95.2% (13.7s) /
elevators πŸ₯‡ 87.7% (0.5s) πŸ₯ˆ 82.6% (6.5s) /
house_16h πŸ₯‡ 67.7% (0.4s) πŸ₯ˆ 52.8% (6.0s) /
house_prices_nominal πŸ₯‡ 89.0% (0.3s) πŸ₯ˆ 78.3% (2.1s) -2.9% (1.2s)
house_sales πŸ₯‡ 89.2% (0.4s) πŸ₯ˆ 77.8% (5.9s) /
mip-2016-regression πŸ₯‡ 59.2% (0.4s) πŸ₯ˆ 34.9% (5.8s) -27.3% (0.4s)
moneyball πŸ₯‡ 93.2% (0.3s) πŸ₯ˆ 91.3% (1.1s) 0.8% (0.2s)
pol πŸ₯‡ 98.7% (0.3s) πŸ₯ˆ 74.9% (4.6s) /
quake -10.7% (0.2s) πŸ₯‡ -1.0% (1.6s) πŸ₯ˆ -10.7% (0.1s)
sat11-hand-runtime-regression πŸ₯‡ 78.3% (0.4s) πŸ₯ˆ 61.7% (2.1s) -56.3% (5.1s)
sensory πŸ₯‡ 29.2% (0.1s) 3.0% (1.6s) πŸ₯ˆ 16.4% (0.0s)
socmob πŸ₯‡ 79.6% (0.2s) πŸ₯ˆ 72.5% (6.6s) 30.8% (0.1s)
space_ga πŸ₯‡ 70.3% (0.3s) πŸ₯ˆ 43.6% (1.5s) 35.9% (0.2s)
tecator πŸ₯ˆ 98.3% (0.1s) πŸ₯‡ 99.4% (0.9s) 78.5% (0.0s)
us_crime πŸ₯ˆ 62.8% (0.6s) πŸ₯‡ 63.0% (2.3s) 6.7% (0.8s)
wine_quality πŸ₯‡ 45.6% (0.2s) πŸ₯ˆ 36.5% (2.8s) 16.4% (1.6s)

Contributing

Prerequisites
1. Set up Git to use SSH
  1. Generate an SSH key and add the SSH key to your GitHub account.
  2. Configure SSH to automatically load your SSH keys:
    cat << EOF >> ~/.ssh/config
    Host *
      AddKeysToAgent yes
      IgnoreUnknown UseKeychain
      UseKeychain yes
    EOF
2. Install Docker
  1. Install Docker Desktop.
3. Install VS Code or PyCharm
  1. Install VS Code and VS Code's Dev Containers extension. Alternatively, install PyCharm.
  2. Optional: install a Nerd Font such as FiraCode Nerd Font and configure VS Code or configure PyCharm to use it.
Development environments

The following development environments are supported:

  1. ⭐️ GitHub Codespaces: click on Code and select Create codespace to start a Dev Container with GitHub Codespaces.
  2. ⭐️ Dev Container (with container volume): click on Open in Dev Containers to clone this repository in a container volume and create a Dev Container with VS Code.
  3. Dev Container: clone this repository, open it with VS Code, and run Ctrl/⌘ + ⇧ + P β†’ Dev Containers: Reopen in Container.
  4. PyCharm: clone this repository, open it with PyCharm, and configure Docker Compose as a remote interpreter with the dev service.
  5. Terminal: clone this repository, open it with your terminal, and run docker compose up --detach dev to start a Dev Container in the background, and then run docker compose exec dev zsh to open a shell prompt in the Dev Container.
Developing
  • This project follows the Conventional Commits standard to automate Semantic Versioning and Keep A Changelog with Commitizen.
  • Run poe from within the development environment to print a list of Poe the Poet tasks available to run on this project.
  • Run poetry add {package} from within the development environment to install a run time dependency and add it to pyproject.toml and poetry.lock. Add --group test or --group dev to install a CI or development dependency, respectively.
  • Run poetry update from within the development environment to upgrade all dependencies to the latest versions allowed by pyproject.toml.
  • Run cz bump to bump the package's version, update the CHANGELOG.md, and create a git tag.