Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

FEA Fused sparse-dense support for PairwiseDistancesReduction #23585

Merged
merged 83 commits into from
Sep 20, 2022
Merged

FEA Fused sparse-dense support for PairwiseDistancesReduction #23585

merged 83 commits into from
Sep 20, 2022

Commits on Jun 13, 2022

  1. MAINT Implement CSR support for all DistanceMetric

    @jjerphan
    jjerphan committed Jun 13, 2022
    Configuration menu
    Copy the full SHA
    b8bd875 View commit details
    Browse the repository at this point in the history

Commits on Jun 14, 2022

  1. Merge branch 'main' into maint/dist-metrics-csr-support

    @jjerphan
    jjerphan committed Jun 14, 2022
    Configuration menu
    Copy the full SHA
    7b07188 View commit details
    Browse the repository at this point in the history

Commits on Jun 15, 2022

  1. TST Remove useless guard

    @jjerphan
    jjerphan committed Jun 15, 2022
    Configuration menu
    Copy the full SHA
    fb99680 View commit details
    Browse the repository at this point in the history

  2. TST Skip JaccardDistance on 32bit architecture

    @jjerphan
    jjerphan committed Jun 15, 2022
    Configuration menu
    Copy the full SHA
    d39d2b2 View commit details
    Browse the repository at this point in the history

Commits on Jun 16, 2022

  1. MAINT Define dtype alias for sparse matrices indices

    @jjerphan
    jjerphan committed Jun 16, 2022
    Configuration menu
    Copy the full SHA
    011e2a2 View commit details
    Browse the repository at this point in the history

  2. MAINT Do not shadow dtype names in Tempita templating

    @jjerphan
    jjerphan committed Jun 16, 2022
    Configuration menu
    Copy the full SHA
    a579630 View commit details
    Browse the repository at this point in the history

  3. fixup! MAINT Define dtype alias for sparse matrices indices

    @jjerphan
    jjerphan committed Jun 16, 2022
    Configuration menu
    Copy the full SHA
    98e9d21 View commit details
    Browse the repository at this point in the history

  4. TST Use cdist and pdist appropriately

    @jjerphan
    jjerphan committed Jun 16, 2022
    Configuration menu
    Copy the full SHA
    8aa4e44 View commit details
    Browse the repository at this point in the history

Commits on Jun 17, 2022

  1. DOC Improve comments 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    9edfa11 View commit details
    Browse the repository at this point in the history

  2. Fixups

    @jjerphan
    jjerphan committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    ee5c6bf View commit details
    Browse the repository at this point in the history

  3. MAINT Wrap of indptr values to support sparse-dense 

    This is kind of an hack for now.

IMO, it would be better to use a flatiter on a view if possible.
See discussions on: https://groups.google.com/g/cython-users/c/MR4xWCvUKHU

Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    bf5eb59 View commit details
    Browse the repository at this point in the history

  4. Apply review comments 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    92b8a6c View commit details
    Browse the repository at this point in the history

  5. More interesting boolean data for tests

    @ogrisel
    ogrisel committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    dc6f8cf View commit details
    Browse the repository at this point in the history

  6. FIX Various corrections

    @jjerphan
    jjerphan committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    bb06f59 View commit details
    Browse the repository at this point in the history

  7. FIX Make Jaccard, Hamming and Hashing robust to explicit zeros 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    a5eb20d View commit details
    Browse the repository at this point in the history

  8. FIX Make the other boolean DistanceMetric also robust to explicit zeros 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    19edf11 View commit details
    Browse the repository at this point in the history

  9. TST Remove xfail for Jaccard on 32bit arch.

    @jjerphan
    jjerphan committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    de86802 View commit details
    Browse the repository at this point in the history

  10. Cast to np.float64_t where appropriate

    @jjerphan
    jjerphan committed Jun 17, 2022
    Configuration menu
    Copy the full SHA
    bb920cf View commit details
    Browse the repository at this point in the history

Commits on Jun 20, 2022

  1. Rename methods and correctly format their signatures 

    Co-authored-by: Christian Lorentzen <lorentzen.ch@gmail.com>
    @jjerphan @lorentzenchr
    jjerphan and lorentzenchr committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    b3759fe View commit details
    Browse the repository at this point in the history

  2. fixup! TST Remove xfail for Jaccard on 32bit arch.

    @jjerphan
    jjerphan committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    7f89236 View commit details
    Browse the repository at this point in the history

  3. FEA CSR support for HaversineDistance

    @jjerphan
    jjerphan committed Jun 20, 2022
    Configuration menu
    Copy the full SHA
    01a0c33 View commit details
    Browse the repository at this point in the history

Commits on Jun 22, 2022

  1. Fix typo 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    7d8a717 View commit details
    Browse the repository at this point in the history

  2. Do not upcast to 64bit yet keep the same precision

    @jjerphan
    jjerphan committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    563e359 View commit details
    Browse the repository at this point in the history

  3. Do use the default rtol

    @jjerphan
    jjerphan committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    f863a51 View commit details
    Browse the repository at this point in the history

  4. Set rtol explicitly in test_distance_metrics_dtype_consistency

    @ogrisel
    ogrisel committed Jun 22, 2022
    Configuration menu
    Copy the full SHA
    5ba0fbe View commit details
    Browse the repository at this point in the history

Commits on Jun 23, 2022

  1. Implement the sparse-dense and the dense-sparse case for c-contiguity 

    Also do test for c-contiguity.
    @jjerphan
    jjerphan committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    4f45839 View commit details
    Browse the repository at this point in the history

  2. Add validation on X and Y, accepting CSR as inputs 

    Co-authored-by: Jérémie du Boisberranger <jeremiedbb@users.noreply.github.com>
    @jjerphan @jeremiedbb
    jjerphan and jeremiedbb committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    3e3e888 View commit details
    Browse the repository at this point in the history

  3. Remove left-overs 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jun 23, 2022
    Configuration menu
    Copy the full SHA
    ddc49d5 View commit details
    Browse the repository at this point in the history

Commits on Jun 24, 2022

  1. Merge branch 'main' into maint/dist-metrics-csr-support

    @jjerphan
    jjerphan committed Jun 24, 2022
    Configuration menu
    Copy the full SHA
    a83887c View commit details
    Browse the repository at this point in the history

Commits on Jul 2, 2022

  1. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Jul 2, 2022
    Configuration menu
    Copy the full SHA
    e8bb70a View commit details
    Browse the repository at this point in the history

  2. Add support for all combinations of {dense,sparse} datasets pairs

    @jjerphan
    jjerphan committed Jul 2, 2022
    Configuration menu
    Copy the full SHA
    dec0aa8 View commit details
    Browse the repository at this point in the history

Commits on Jul 5, 2022

  1. Const-qualify X and Y

    @jjerphan
    jjerphan committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    63c6fe3 View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    0bb368f View commit details
    Browse the repository at this point in the history

  3. Only pass Y_norm_squared for the dense-dense case

    @jjerphan
    jjerphan committed Jul 5, 2022
    Configuration menu
    Copy the full SHA
    30e84af View commit details
    Browse the repository at this point in the history

Commits on Jul 17, 2022

  1. Update comments

    @jjerphan
    jjerphan committed Jul 17, 2022
    Configuration menu
    Copy the full SHA
    780d7bb View commit details
    Browse the repository at this point in the history

Commits on Jul 19, 2022

  1. Pop unused keywords arguments

    @jjerphan
    jjerphan committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    72f4ae7 View commit details
    Browse the repository at this point in the history

  2. Remove unused import

    @jjerphan
    jjerphan committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    a1ce042 View commit details
    Browse the repository at this point in the history

  3. DOC Add whats_new entry

    @jjerphan
    jjerphan committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    713b932 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    ac6208d View commit details
    Browse the repository at this point in the history

  5. fixup! Pop unused keywords arguments

    @jjerphan
    jjerphan committed Jul 19, 2022
    Configuration menu
    Copy the full SHA
    5570f71 View commit details
    Browse the repository at this point in the history

Commits on Jul 20, 2022

  1. DOC Update comment and changelog 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Jul 20, 2022
    Configuration menu
    Copy the full SHA
    4c455ea View commit details
    Browse the repository at this point in the history

Commits on Jul 21, 2022

  1. MAINT Test second alternative for sparse-dense support 

    The modulo trick is costly because it uses integer division
under the hood which are up to 80 times slower.

An alternative is to pass the address of the `indices` array and shift
this address at the callers level with the information that we have
there so that the dereferencing works properly.

This commit replaces the modulo trick by this alternative and adapt
documentation and comments in this regard.

See:
scikit-learn#23585 (comment)
    @jjerphan
    jjerphan committed Jul 21, 2022
    Configuration menu
    Copy the full SHA
    0f0ea70 View commit details
    Browse the repository at this point in the history

Commits on Aug 8, 2022

  1. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    80b8c02 View commit details
    Browse the repository at this point in the history

  2. DOC Update comment and changelog 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Aug 8, 2022
    Configuration menu
    Copy the full SHA
    a3cf4d8 View commit details
    Browse the repository at this point in the history

Commits on Aug 9, 2022

  1. Merge branch 'maint/pdr-sparse-support' into alt/feat/pdr-sparse-support

    @jjerphan
    jjerphan committed Aug 9, 2022
    Configuration menu
    Copy the full SHA
    e9da38b View commit details
    Browse the repository at this point in the history

Commits on Aug 10, 2022

  1. DOC Improve comments and code self-documentation 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    e9ecbbc View commit details
    Browse the repository at this point in the history

  2. Merge branch 'main' into maint/pdr-sparse-support 

    This update the branch after the merge of scikit-learn#23865.
    @jjerphan
    jjerphan committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    c8bacc6 View commit details
    Browse the repository at this point in the history

  3. Merge branch 'maint/pdr-sparse-support' into alt/feat/pdr-sparse-support

    @jjerphan
    jjerphan committed Aug 10, 2022
    Configuration menu
    Copy the full SHA
    180a54e View commit details
    Browse the repository at this point in the history

Commits on Aug 11, 2022

  1. TST dtype-parametrize test_format_agnosticism

    @jjerphan
    jjerphan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    de52371 View commit details
    Browse the repository at this point in the history

  2. fixup! TST dtype-parametrize test_format_agnosticism

    @jjerphan
    jjerphan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    3243153 View commit details
    Browse the repository at this point in the history

  3. MNT Pushing data up instead of indices

    @thomasjpfan
    thomasjpfan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    e521992 View commit details
    Browse the repository at this point in the history

  4. DOC Improve comment

    @thomasjpfan
    thomasjpfan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    972fff9 View commit details
    Browse the repository at this point in the history

  5. REV Revert back to memoryviews for indices

    @thomasjpfan
    thomasjpfan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    3086c0b View commit details
    Browse the repository at this point in the history

  6. DOC Spelling

    @thomasjpfan
    thomasjpfan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    afa0c35 View commit details
    Browse the repository at this point in the history

  7. Merge pull request #16 from thomasjpfan/alt/feat/pdr-sparse-support 

    MNT Pushing data up instead of indices
    @jjerphan
    jjerphan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    bc78747 View commit details
    Browse the repository at this point in the history

  8. Merge pull request #15 from jjerphan/alt/feat/pdr-sparse-support 

    MAINT Division-less alternative for fused sparse-dense support
    @jjerphan
    jjerphan committed Aug 11, 2022
    Configuration menu
    Copy the full SHA
    bd48ef0 View commit details
    Browse the repository at this point in the history

Commits on Aug 22, 2022

  1. TST Suggest logic adaptation for _pairwise_{dense_sparse,sparse_dense} 

    This fixes those two methods used for testing which were using
the previous trick.

This allocate potentially large `indices` arrays.

This also add missing kwargs for some others tests.
    @jjerphan
    jjerphan committed Aug 22, 2022
    Configuration menu
    Copy the full SHA
    be59297 View commit details
    Browse the repository at this point in the history

Commits on Aug 25, 2022

  1. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    f8ab496 View commit details
    Browse the repository at this point in the history

  2. DOC Add co-authors in whats_new entry 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
Co-authored-by: Thomas J. Fan <thomasjpfan@gmail.com>
    @jjerphan @ogrisel @thomasjpfan
    3 people committed Aug 25, 2022
    Configuration menu
    Copy the full SHA
    ec1d4f9 View commit details
    Browse the repository at this point in the history

Commits on Aug 26, 2022

  1. Do not support CSR matrices without non-zero elements

    @jjerphan
    jjerphan committed Aug 26, 2022
    Configuration menu
    Copy the full SHA
    fbf311e View commit details
    Browse the repository at this point in the history

Commits on Aug 28, 2022

  1. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Aug 28, 2022
    Configuration menu
    Copy the full SHA
    511c6e6 View commit details
    Browse the repository at this point in the history

  2. fixup! Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Aug 28, 2022
    Configuration menu
    Copy the full SHA
    8fddffd View commit details
    Browse the repository at this point in the history

Commits on Aug 29, 2022

  1. MAINT Do not pop Y_norm_squared when unused

    @jjerphan
    jjerphan committed Aug 29, 2022
    Configuration menu
    Copy the full SHA
    4b879f1 View commit details
    Browse the repository at this point in the history

Commits on Sep 9, 2022

  1. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    0b1ce13 View commit details
    Browse the repository at this point in the history

  2. Explicitly do not support CSR matrices with int64 indices and indptr 

    See: scikit-learn#23653
    @jjerphan
    jjerphan committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    ca49236 View commit details
    Browse the repository at this point in the history

  3. DOC Update and improve comment for the alternative CSR representation

    @jjerphan
    jjerphan committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    d7b3649 View commit details
    Browse the repository at this point in the history

  4. fixup! DOC Update and improve comment for the alternative CSR represe… 

    …ntation
    @jjerphan
    jjerphan committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    5e13663 View commit details
    Browse the repository at this point in the history

  5. CI Retrigger CI due to faulty runs

    @jjerphan
    jjerphan committed Sep 9, 2022
    Configuration menu
    Copy the full SHA
    1de8acb View commit details
    Browse the repository at this point in the history

Commits on Sep 12, 2022

  1. Merge branch 'main' into maint/pdr-sparse-support

    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    7766388 View commit details
    Browse the repository at this point in the history

  2. DOC Update whats_new entry

    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    8f43a5a View commit details
    Browse the repository at this point in the history

  3. Test and document Isomap on sparse data

    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    a229b35 View commit details
    Browse the repository at this point in the history

  4. Test and document TSNE on sparse data

    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    3e357a2 View commit details
    Browse the repository at this point in the history

  5. Test and document pairwise_distances_argmin on sparse data

    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    9723115 View commit details
    Browse the repository at this point in the history

  6. Test and document LocalOutlierFactor on sparse data

    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    faf704a View commit details
    Browse the repository at this point in the history

  7. DOC Add support for sparse data for NearestNeighbors, KNeighbors*, Ra… 

    …diusNeighbors* methods
    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    c66bb82 View commit details
    Browse the repository at this point in the history

  8. DOC Remove formatting change

    @jjerphan
    jjerphan committed Sep 12, 2022
    Configuration menu
    Copy the full SHA
    1eb5b2c View commit details
    Browse the repository at this point in the history

Commits on Sep 15, 2022

  1. TST Do not test on full cartesian product 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    fcf15b6 View commit details
    Browse the repository at this point in the history

  2. fixup! TST Do not test on full cartesian product

    @jjerphan
    jjerphan committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    58453d7 View commit details
    Browse the repository at this point in the history

  3. TST Add TODO for consistency checks on results for sparse and dense data 

    Co-authored-by: Olivier Grisel <olivier.grisel@ensta.org>
    @jjerphan @ogrisel
    jjerphan and ogrisel committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    63fda8c View commit details
    Browse the repository at this point in the history

  4. MAINT Mark PairwiseDistancesReductions as unusable for some config.

    @jjerphan
    jjerphan committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    1d7bcc7 View commit details
    Browse the repository at this point in the history

  5. fixup! MAINT Mark PairwiseDistancesReductions as unusable for some co… 

    …nfig.
    @jjerphan
    jjerphan committed Sep 15, 2022
    Configuration menu
    Copy the full SHA
    fec55bf View commit details
    Browse the repository at this point in the history

Commits on Sep 20, 2022

  1. TST Improve test_format_agnosticism 

    …and add a space somewhere for proper formatting.

Co-authored-by: Meekail Zain <Micky774@users.noreply.github.com>
    @jjerphan @Micky774
    jjerphan and Micky774 committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    d55bcec View commit details
    Browse the repository at this point in the history

  2. DOC Update comment regarding the use of pairwise_distances_chunked 

    See: `BaseDistanceReductionDispatcher.valid_metrics`

Co-authored-by: Meekail Zain <Micky774@users.noreply.github.com>
    @jjerphan @Micky774
    jjerphan and Micky774 committed Sep 20, 2022
    Configuration menu
    Copy the full SHA
    c21187a View commit details
    Browse the repository at this point in the history