Start building and publishing conda nightlies

[charlesbluca]

Should unblock the issues around pinning described in dask/distributed#8575 (comment)

[fjetter]

dask-expr 1.0.4+6.gd7238e9 has requirement dask==2024.3.1, but you have dask 2024.3.0+3.gf9310c440. Is this a chicken egg problem where we have to merge on failing build to let the other pass or what is the expected behavior?

[bdice]

@fjetter I discussed this a bit with @charlesbluca. It looks like dask-expr’s pip check cannot pass because dask is versioned incorrectly. The conda package is versioned as an alpha of 2024.3.1, but the internal versioneer version is a post-release of 2024.3.0. This would have to be aligned in dask to make this check pass.

The options are probably:

• don’t add 1 to the patch release in the conda recipe, match the versioneer post-release versioning (make conda match versioneer)
• hack the versioneer versioning to add 1 to the patch and make it an alpha (make versioneer match conda)
• turn off pip check

[jrbourbeau]

Thanks @charlesbluca. Btw, a commit just got pushed to dask/dask, so there should be a new nightly build uploaded shortly which should fix CI here (based off our offline conversation earlier today)

[charlesbluca]

Thanks @jrbourbeau! Just pulled the latest changes to retrigger CI, looks like that resolved the conda issues 🎉

The conda package is versioned as an alpha of 2024.3.1, but the internal versioneer version is a post-release of 2024.3.0.

This is correct, and I filed dask/community#369 to track efforts to make these versions consistent.

This would have to be aligned in dask to make this check pass.

I don't think this is the case, I think the main reason the check failed before (as opposed to now) is that before the latest dask-core nightlies we were picking up were 2024.3.1a that preceded the last release (hence the internal 2024.3.0+... version). Now that a commit has been pushed to dask/dask since the last release (dask/dask@8c2199e) to trigger a nightly build, we now have2024.3.2a nightlies available with the internal 2024.3.1+... version that dask-expr is looking for ("dask-expr 1.0.4+6.gd7238e9 has requirement dask==2024.3.1...").

That being said, we shouldn't have had to wait for a random commit to land in the repo post-release to unblock things here, which is why I opened dask/dask#11014 so that, moving forward, the normal release process of pushing a tag to dask/dask will trigger a nightly build/publish and resolve this blocker immediately.

[rjzamora]

@jrbourbeau - Any objections to getting this in?

[bdice]

Yes, @charlesbluca is right. Thanks for the correction!

[jrbourbeau]

Any objections to getting this in?

No objection from me generally.

That said, I think we still need to add the conda token as a repo secret for uploading, right?

EDIT: Or is that handled at a higher level, like the dask org?

[charlesbluca]

EDIT: Or is that handled at a higher level, like the dask org?

My understanding from this comment when we first set up nightlies is that it was set at the org level?

dask/dask#8469 (comment)

[rjzamora]

Thanks @charlesbluca!

[rjzamora]

Looks like you were correct @jrbourbeau: https://github.com/dask/dask-expr/actions/runs/8426623516/job/23075334138

[rjzamora]

Okay yeah, @charlesbluca shared this link with me offline: https://docs.github.com/en/actions/security-guides/using-secrets-in-github-actions#creating-secrets-for-an-organization

From the Repository access dropdown list, choose an access policy.

Seems like dask-expr needs to be granted access from a dropdown list at the org level.

[quasiben]

I updated the DASK_CONDA_TOKEN to be shared with the dask-expr repo

[charlesbluca]

Thanks @quasiben! Looks like that unblocked publishing nightlies:

https://github.com/dask/dask-expr/actions/runs/8437562794/job/23107649428
https://anaconda.org/dask/dask-expr/files

I've opened dask/distributed#8601 to apply the necessary changes to the dask nightly recipe, which supersedes dask/distributed#8575 and should unblock Distributed's conda CI.

[jrbourbeau]

Great, thanks @charlesbluca @rjzamora @quasiben for handling this