GBM Prediction Interval

[AlexanderKUA]

I need to get prediction interval for GBM model (loss='ls'). I'm using this example as a basis

http://scikit-learn.org/stable/auto_examples/ensemble/plot_gradient_boosting_quantile.html

My model parameters are chosen by GridSearch, than I'm training with the same params 2 models for lower and higher bounds.
But my Prediction curve sometimes is outside of Prediction interval. It looks odd. How can I solve such issue?

[amueller]

ping @pprett

[raghavrv]

Could you post a minimal code ? If its too long use gists

[AlexanderKUA]

No. Unfortunately it's part of complex system. I'll try to find equivalent example.

[pprett]

@AlexanderKUA you are saying that your loss='ls' model is outside the prediction interval? There might be multiple reasons: a) model is a poor approximation for the quanitles/mean or b) your data is heavily skewed, then the mean might be indeed outside the prediction interval. Can you show a histogram of your target variable?

[AlexanderKUA]

I made histograms(and examples of appropriate prediction intervals) of my target variable (please take into account that it is in range [0, 100]).

case #2

case #3

case #6

As you can see with case6(last) everything looks ok, but target variable distribution is also skewed as in previous(case2, case3).

Thanks in advance

[lorentzenchr]

One possibility is to estimate quantiles conditional on X, implemented in #924. While this won't give true prediction intervals as the estimation error is not accounted for, it gives at least some sort of uncertainty.