[WIP] next release (3.0.0)

[guolinke]

This is to continue #3071, we have several new items that are not finished.
I plan to release the 3.0.0 by the end of Sep.

PRs:

• [R-package] added support for first_metric_only (fixes #2368) #2912
• Fix add features  #2754
• Add CTR conversion for categorical features (with bundling support) #2463
• Less restrictive monotone constraints #2305 (last remained part)

New Features

• GPU support on ppc64le #2937
• Unify out results of LGBM_BoosterDumpModel and LGBM_BoosterSaveModel #2604
• Add some features to R-package that are already in C API and Python package (Add support for saving weight values of a node in the R-package #2281, [R-package] exclude training data from being checked for early stopping #2472, Refit method for R-package #2369, [R-package] Add support for non-ASCII feature names #2983)
• CRAN release of course [R-Package] CRAN issues #629

Fixes (may is not included in 3.0.0):

• All known GPU bugs (lightgbm.basic.LightGBMError: Bug in GPU histogram! split 11937: 12, smaller_leaf: 10245, larger_leaf: 1704 #2793, [gpu] /root/repo/LightGBM/compute/include/boost/compute/utility/wait_list.hpp:166: void boost::compute::wait_list::wait() const: Assertion `clWaitForEvents(size(), get_event_ptr()) == 0' failed. Aborted (core dumped) #2648, OpenCL Bus error #2550, GPU build fails for lightgbm target #2126, Training on GPU fails (OSError: exception: access violation) #1717, BUG in GPU histogram #1003)
• MMLSpark issues([mmlspark] Memory Corruption #2392, JVM crash caused by LGBM_DatasetCreateFromCSRSpark #2363)

[guolinke]

cc @shiyu1994 for CTR features (and symmetric trees).

[jameslamb]

@guolinke sounds good! I think that to get to CRAN, I really need the help of you and other C++ maintainers on #3187. I'm trying to get better at C++ but it's beyond my ability to fix that issue by September.

I also think we should attempt a submission to CRAN right now (of 3.0.0-1). I think it's very unlikely that it will be accepted without #3187, but through that submission we might learn of other things that need to be fixed.

See this comment: #3187 (comment)

[guolinke]

@jameslamb no problem. sorry for missing #3187, I will try to submit it to cran.
@shiyu1994 can you help with the 32-bit build problem?

[guolinke]

@jameslamb
The submit results:

[shiyu1994]

@jameslamb no problem. sorry for missing #3187, I will try to submit it to cran.
@shiyu1994 can you help with the 32-bit build problem?

Ok.

[StrikerRUS]

@guolinke I believe RC candidates are for beta testing features that will be in the corresponding ordinary release. No new features are allowed after RC is released, only bug fixes reported during beta testing period. All features you've mentioned in your staring comment should be included in 3.1.0 version as they are non-breaking ones.

ping @henry0312 to confirm

[jameslamb]

@jameslamb
The submit results:

woo, thanks so much! You will get an email some time in the next 48 hours with requested changes, and links to logs.

You could share those on #629 or in Slack, so we don't clutter this issue.

[aakarshg]

hello, can we also add #3306 to the 3.0.0 release?

[StrikerRUS]

@aakarshg

hello, can we also add #3306 to the 3.0.0 release?

Hi!
I'm afraid that adding new feature will require one more round of testing, i.e RC2. I believe it'll be better to release 3.0.1 or 3.1.0 ASAP with merged all PRs proposing new features which all currently open and (almost) approved.

[StrikerRUS]

@guolinke Please share your thoughts about #3293 (comment) and #3293 (comment)

[guolinke]

@StrikerRUS
My purpose to release the 3.0.0rc1, is to have a time buffer to include some on-going new features into 3.0.0 .
If the next version is 3.1.0, I should release the 3.0.0 directly..

[jameslamb]

I agree, the next release should be 3.0.0 and we should do it ASAP. That was the purpose of having a release candidate.

[StrikerRUS]

@guolinke

My purpose to release the 3.0.0rc1, is to have a time buffer to include some on-going new features into 3.0.0 .

I thought that 3.0.0rc1 and 3.0.0 should be identical in terms of functionality. But I might be wrong.

[StrikerRUS]

It looks like early adopters of LightGBM have been using code from master branch and ordinary users are not aware of RC and how to install it. Given that, I think we can release 3.0.0 next week or week after week. WDYT guys?

[jameslamb]

I agree with @StrikerRUS . Especially since CRAN maintainers will be back on Monday 😀

[guolinke]

@jameslamb @StrikerRUS I think we can release the 3.0.0 after R's logo ready.

[jameslamb]

It's ok with me if we release before the logo is ready for R! Since that change only affects our documentation and can be done without needing to re-release the package.

[StrikerRUS]

What do you think about the try to submit to CRAN before the release? I believe it will be great to sync releases.

[guolinke]

yeah, it is a good idea. @jameslamb could you provide me a package to submit?

[jameslamb]

yep! I decided to take this opportunity to start working on #3283 ... just made #3335 with a first effort for that.

This package has the version changed to 3.0.0 and can be submitted to CRAN!

lightgbm_3.0.0.tar.gz

[guolinke]

@jameslamb I am waiting for the cran's reply. it has been the whole day.
Once the cran got accepted, we can release 3.0.0 .

[jameslamb]

great! I bet they will be slow this week, working through the backlog from vacation.

Also the delay might mean we finally made it past the automated checks and that the package is being reviewed by a human 😎

[guolinke]

@jameslamb finally get a reply.

Dear maintainer,

package lightgbm_3.0.0.tar.gz has been auto-processed and is pending a manual inspection of this new CRAN submission. A CRAN team member will typically respond to you within the next 10 working days. For technical reasons you may receive a second copy of this message when a team member triggers a new check.

Log dir: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200825_035150%2F&data=02%7C01%7Cguolin.ke%40microsoft.com%7Cc89735e17a834f1480dc08d84a5aab4b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637341101018660880&sdata=V1GESHLP8Pke0469avFLaDusUkFv9gpcOgrGZm9hUKA%3D&reserved=0
The files will be removed after roughly 7 days.
Installation time in seconds: 668
Check time in seconds: 265
R Under development (unstable) (2020-08-23 r79071)

Pretests results:
Windows: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200825_035150%2FWindows%2F00check.log&data=02%7C01%7Cguolin.ke%40microsoft.com%7Cc89735e17a834f1480dc08d84a5aab4b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637341101018660880&sdata=9JxAU5uj87f9yRxUj3NHkRI1GXg%2BZ9HyvjhkHwAmamI%3D&reserved=0
Status: 1 NOTE
Debian: https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwin-builder.r-project.org%2Fincoming_pretest%2Flightgbm_3.0.0_20200825_035150%2FDebian%2F00check.log&data=02%7C01%7Cguolin.ke%40microsoft.com%7Cc89735e17a834f1480dc08d84a5aab4b%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637341101018660880&sdata=lCsxRM6ujhIZ61gw2HZDjOQnm3ssSAvl8AJeDNvPdRk%3D&reserved=0
Status: 1 NOTE

No strong reverse dependencies to be checked.

Best regards,
CRAN teams' auto-check service

[guolinke]

and

Thanks, we see:

License components with restrictions and base license permitting such:
MIT + file LICENSE
File 'LICENSE':
The MIT License (MIT)

 Copyright (c) Microsoft Corporation

 Permission is

Please only ship the CRAN template fior the MIT license.

Is there some reference about the method you can add in the Description field in the form Authors (year) doi:.....?

Please fix and resubmit.

[jameslamb]

oooo I know how to fix those!

For the license, we need to do this (https://github.com/tidyverse/tidytemplate/blob/master/LICENSE) instead of having the entire license text (https://github.com/microsoft/LightGBM/blob/master/R-package/LICENSE). They do that to make it easier to do automated checks and to prevent people from saying their project follows a mainstream license but then sneaking in other changes.

For the doi, I think we can link to the original LightGBM paper like this:

• https://github.com/cran/glmnet/blob/master/DESCRIPTION#L20
• https://github.com/cran/MVA/blob/master/DESCRIPTION#L13

So we can add a link like to http://papers.nips.cc/paper/6907-lightgbm-a-highly-efficient-gradient-boosting-decision and a citation in the description. I can submit a PR with a new package to upload in an hour or so.

[StrikerRUS]

3.0.0 has been released in all distribution channels that we officially support, except CRAN. I believe we can close this issue because we have a separate issue for CRAN.

[jameslamb]

agreed!

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. To start a new related discussion, open a new issue at https://github.com/microsoft/LightGBM/issues including a reference to this.