using scikit learn regression

[tesla-cat]

I am trying to learn scikit-learn regression using a simple function shown below:

import matplotlib.pyplot as plt
import numpy as np
from sklearn.ensemble import GradientBoostingRegressor, RandomForestRegressor
from sklearn.gaussian_process import GaussianProcessRegressor
from sklearn.gaussian_process.kernels import RBF
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split
from sklearn.multioutput import MultiOutputRegressor
from sklearn.neural_network import MLPRegressor
from sklearn.svm import SVR


def func(x: np.ndarray):
    # x (N, 24), y (N, 2)
    x_mean = np.mean(x, axis=1)
    a = np.sin(x_mean)
    b = np.cos(x_mean)
    y = np.array([a, b]).T
    return y


np.random.seed(0)
x = np.random.rand(1000, 24) * np.pi * 10
y = func(x)

x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=0)

models = [
    GradientBoostingRegressor(),
    RandomForestRegressor(),
    GaussianProcessRegressor(RBF()),
    MLPRegressor(max_iter=1000),
    SVR(),
]

plt.figure(figsize=(5, len(models) * 3))

for i, base_model in enumerate(models):
    name = base_model.__class__.__name__
    model = MultiOutputRegressor(base_model)
    model.fit(x_train, y_train)
    y_pred = model.predict(x_test)
    mse = mean_squared_error(y_test, y_pred)

    x_mean = np.mean(x_test, axis=1)
    plt.subplot(len(models), 1, i + 1)
    title = f"{name}. MSE: {mse:.2e}"
    print(title)
    plt.title(title)
    plt.plot(x_mean, y_test[:, 0], ".", label="y_test[:, 0]")
    plt.plot(x_mean, y_test[:, 1], ".", label="y_test[:, 1]")
    plt.plot(x_mean, y_pred[:, 0], ".", label="y_pred[:, 0]")
    plt.plot(x_mean, y_pred[:, 1], ".", label="y_pred[:, 1]")
    plt.legend()

plt.tight_layout()
plt.savefig("regression_test.png")

but the results are not good:

I think I am not using these regressors correctly.

How should I modify my code?

update

by reducing the dim of x, I get some good results:

how should I change the code for GP to work with high dim input?

[jmarintur]

Dear @tesla-cat, although it may seem a simple function, it could be challenging. As you see, when you simply use one dimension, x_mean is equal to x, decreasing the complexity of the problem, that's why you are getting better results. When you increase dimensions, you are going to end up having multiple feature vectors with the same y, due to the fact that you apply the mean and then on top periodic functions to get a and b. I suggest you start playing with values like these:

x = np.random.rand(1000, 4) * np.pi * 2

This way you will get the following results:

[tesla-cat]

Hi friend, thank you for the reply.

In fact I have tried different dims of x as shown in my last image

This is actually a sanity check problem before dealing with a real-world problem:

ML of financial data where there are 24 observations (technical indicators), the financial data is extremely noisy.

[jmarintur]

Then, perhaps you should try with different stats, e.g. mean, std, max, min, etc on top of your raw features. You could also use the original feature vector as input, x, and even normalising your data.