Suggested path to a 5.0 RC

[joelostblom]

@jakevdp @ChristopherDavisUCI @mattijn I made this so that it is easier for us to keep track of what needs to be done before a release candidate for Altair 5 is published. This is just my understanding of what is left from skimming the open issues, so feel free to suggest changes. Once we have a release candidate out, I can go through all open issues in more detail to close everything that works in the RC.

Important

I think we should at minimum address these five issues before a release candidate is created.

• DOCS: Revamp docs for the 5.0 release #2566 (merged)
• MAINT: Make sure that deprecation warnings are displayed #2577 (merged)
• Lifting parameters to the top level #2702 (merged)
• MAINT: Update from v4 to v5 in some additional files #2582 (merged)
• MAINT: Update examples and tests to VL5 syntax #2576 (merged)
• parameters added to a layer should be serialised outside the scope of the layer #2597 (merged)
• Add tests for all new VegaLite functionality since 4.17 #2703 (merged)
• Update release notes with all the PRs since the last release #2836 (merged)
• include underscore after view #3030 (merged)
• naming parameter views of layered repeated charts + interactivity #2849 (merged)

Dream come true

Technically not blocking but I am 😍 for this issue. I think it is more attractive than the "nice-to-haves", and more important to make a decision on before the 5.0 release since it would be preferred to introduce such as big addition in a major release.

• Add method-based attribute setting #2795 (merged)

Nice to have

These are not blocking and could be part of minor 5.x releases.

• Remove broken Wikipedia donations chart #2625 (merged)
• Equal default chart dimensions #2581 (merged)
• Chart representation changes after the chart is displayed #2497 (merged)
• ENH: Make the schema validation error for non-existing params more informative #2568 (merged)
• Represent pandas ordered categoricals as ordinal data #2522 (merged)
• Hide deprecated callables from code completion suggestions #2814 (merged)
• Remove deep validation and instead use error hierarchy to improve error messages #2842 (merged)
• Update parameter docstrings and signatures #2908 (merged)
• Check all specs in the context of top level data #2835 (not planned)
• Support Vega-Lite's optional encoding types #2584 (not planned)
• FEAT: Include only referenced data columns in chart specs #2586 (not planned)
• Enhancement of Vega-Embed CSS for Improved Display and Flexibility #2867 (merged)
• Add simpler ways of controlling element and font size #2820 (not started)

Docs (these could come at a later point as well, but would be great to sync with a big release):

• Improve documentation on geographical visualizations #2580 (merged)
• Include documentation for each mark type #2578 (merged)
• Document the differences between value and datum? #2572 (merged)
• Update resource section #2415 (merged)

Downstream dependent packaging issues

• Archiving v4 documentation of Altair #2865 (done) [altair-viz-v4]
• Update entrypoint to Vega-Lite 5 so Altair v5 chart renders altair-viz/altair_viewer#52 (done) [altair_viewer]
• Mimetype renderer not working #2585 (not started) [jupyter-lab]
• Make altair_viewer and altair_saver work with Altair 5 #2866 (not started) [altair_viewer & altair_saver]

Upstream dependent VL issues

The VL-specification are created like we anticipated for the compound charts in Altair, but there are a few outstanding issues for rendering on the VL-side:

• duplicate parameter definition error in layered charts (because of "point": true) vega-lite#7854
• repeat operator with defined width and height does not accept layer vega-lite#8523
• repeat chart with top level defined brush selection corrupts cross-view interaction vega-lite#8348
• chart with repeat and a defined name disables the independency of the repeated channel (eg y-axis) vega-lite#8446

graph LR;
  vega-lite:::doneok--"Open VL issues (0)" -->altair
  subgraph altair
    svlg--"Open issues (1)"--> slvg_done 
    thef--"Open issues (0)"--> thef_done:::doneok
    ds--"Open issues (0)"--> ds_done:::doneok 
    direction LR
    subgraph svlg ["support VL grammar"]
    end
    subgraph slvg_done ["all done"]
    end
    subgraph thef ["type hinting & error feedback"]
    end
    subgraph thef_done ["all done"]
    end
    subgraph ds ["new docs & site"]
    end
    subgraph ds_done ["all done"]
    end
  end
  subgraph altair-viz-v4 ["altair-viz-v4 \narchived v4 docs"]
    end
  altair--"Open issues (0)"-->altair_viewer:::doneok
  altair--"Open issues (1)"-->altair_saver
  altair--"Open issues (0)"-->jupyter-lab:::doneok
  altair--"Open issues (0)"-->altair-viz-v4:::doneok
  classDef doneok stroke:#3C793E,fill:#53A158,fill-opacity:0.5

[joelostblom]

@jakevdp is there anything I can do to help with the review of the open PRs and the progress towards the release of the next Altair version? I would be happy to co-maintain the library if you are comfortable with me reviewing and merging low risk PRs, such as simpler fixes (e.g. #2576, #2577, #2582) and those related to the docs (e.g. #2566, #2607, and #2610).I think we have some momentum currently with multiple people contributing and it would be great if we could sustain that as much as possible while not overloading you with things to review, but I understand that this can be a big decision so I am happy to listen to other ideas of how I can be helpful here.

@mattijn maybe you would also be interested to help out with PR review/merging too if that is an option?

[mattijn]

Yes, yes!
Would be a pity if the master branch get stale and simultaneously too much adopted in current condition.

[ChristopherDavisUCI]

That sounds like a great idea to me, if @joelostblom and @mattijn could help to merge some of the more modest PRs. (I'd offer to help myself but I don't understand git/GitHub well enough.) I'm very interested in working on some updates (especially over the summer) but I've found it difficult to keep track of the not-yet-merged work. In general @jakevdp if there's anything I can do to help, please just let me know and I'll be very happy to try. (As a side note, I'm excited to see new people contributing recently!)

[mattijn]

As was pointed out in #2686 (comment). Once Altair 5 RC is released altair_viewer and altair_saver won't work anymore as they are using the entrypoints of altair vegalite 4.

Observe current behaviour using altair from GitHub main branch:

import altair as alt
from vega_datasets import data
import altair_viewer

cars = data.cars.url

op_var = alt.param(value=0.1)

chart = alt.Chart(cars).mark_circle(opacity=op_var).encode(
    x='Miles_per_Gallon:Q',
    y='Horsepower:Q',
    color='Origin:N'
).add_params(
    op_var
)

altair_viewer.display(chart)

---------------------------------------------------------------------------
NoMatchingVersions: No matches for version='5.2.0' among ['4.0.2', '4.8.1', '4.17.0'].
Often this can be fixed by updating altair_viewer:
    pip install -U altair_viewer

[mattijn]

I just noticed the last checkbox in the OP, that describes already that these companion packages require updating.

[ChristopherDavisUCI]

@joelostblom @mattijn Is there anything in particular that I could focus on to help with this release?

[binste]

Thanks mattjin for the feedback here and in the open PRs and issues! I'll have a look.

I'm also in favor of archiving the v4 docs. For reference, we should add something similar to the "This docs is for version 4" banner at the top of https://vega.github.io/vega-lite-v4/. Code at https://github.com/vega/vega-lite-v4/blob/master/_layouts/plain.html#L7 On the new page we could try to add a version switcher to make it easy to get to the v4 docs.

[ChristopherDavisUCI]

Thanks @mattijn, great to see all your progress! I took a look at what you linked for the docs, but it's all pretty incomprehensible to me unfortunately. I assume you've seen this page that gives some (also not very comprehensible to me) instructions related to the docs and releasing: https://github.com/altair-viz/altair/blob/master/RELEASING.md

(I also wasn't sure what specific part you were asking about... was it more about getting v5 published, or more about how to archive v4?)

Maybe @binste would have more helpful feedback on this?

[binste]

The altair documentation is hosted using github pages as an "Organization site". This means that the repository https://github.com/altair-viz/altair-viz.github.io contains the source files for the website. If you push changes to those files, they are automatically deployed and visible at https://altair-viz.github.io. I'm doing the same to host a development version of the docs at https://binste.github.io/altair-docs/index.html

Before running sync_website.sh, one needs to build the documentation locally as described in RELEASING.md. Then, the script will:

• Clone https://github.com/altair-viz/altair-viz.github.io
• Replacing the files for the current documentation with the new documentation. You'll need to make sure that the .nojekyll file still is in the docs repository after this step as it does not come from sphinx but needs to be added manually. Not sure if the script respects this or will delete it
• Pushing the new files to GitHub where they will be automatically deployed

To create an archived version of the v4 docs, it would probably look something like this:

• Create a new repo called for example "altair-viz-v4"
• Copy over current files from the https://github.com/altair-viz/altair-viz.github.io repo (including the .nojekyll file), commit and push
• Go to the repo settings and set up GitHub Pages as described in step 4 for "Project site" here
• Page should be live within a few minutes at https://altair-viz.github.io/altair-viz-v4

Hope this is helpful! Happy to assist with any of these steps if needed.

[mattijn]

This is really helpful @binste. Thanks for making this really clear.
@jakevdp, do you agree with this as well? Since we don't have the rights, we were wondering if you could assist us in this process of setting up another repo within the altair-viz organisation page.

[mattijn]

Regarding the documentation. I'm fine to publish the v5 altair release page together with the altair 5.0rc1. Maybe we can add an announcement header there as well just like the archive-page of the v4-docs.

[mattijn]

@binste, I saw your comment regarding an official Altair release here: #2770 (comment). So I thought I add some more info here:

Since we have adopted this top-level parameter strategy, there are still a few outstanding issues on the VL side which will currently cause regression on the Altair side. So I think we have to wait for a new release on the VL side (v5.7.0?), before we can do an official v5.0 release of Altair.

For a release candidate version of Altair (v5.0rc1) I'm willing to accept these discrepancies and publish against Vega-Litev5.2.0 since that version of Vega-Lite is currently available in the whole Altair family of packages (altair, altair_saver, altair_viewer).

The outstanding VL-issues are currently summarized in this comment #2702 (comment).
These can be reduced to the following one/two issues, which I think are the main bottleneck:

• duplicate parameter definition error in layered charts (because of "point": true) vega-lite#7854 (layered chart with point=True, very common syntax)
• repeat chart with top level defined brush selection corrupts cross-view interaction vega-lite#8348 (maybe not so common in practice, but one of the main selling-charts of VL)

And since it is currently holiday season we hope that the people involved can find some relaxing code time;)

After there is a new release of Vega-Lite, there are a few repositories within the altair-viz family that require a new release. That will take, with a bit of luck, one or two weeks.

[binste]

I'm wondering if any vega-lite maintainers are aware of which issues block the Altair 5 release. I don't want to pressure anyone, just thinking about creating an issue in the vega-lite repo which summarises the open points so it's easier to track for them and hopefully resolve. Are you aware of such a summary? Happy to create one based on the information from @mattijn provided above. Or is anyone in direct contact with a vega-lite maintainer?

[mattijn]

I've had contact with @arvind before and he is aware. If we can assist directly with the Vega-lite repositroy to resolve these issues then that would be great too and I'm sure much appreciated. Without speaking for anyone, people grow into other roles and are scrutinised in the publish or perish / tenure adventure, together with private obligations, time becomes scare for open-source maintenance work, but simultaneously are much delighted to see our work.
Shall we give it a try to fix an issue in the Vega-lite repository? It means that we, people with a python background, must go out of our comfort zone. Technically speaking as far as I can see, this issue, vega/vega-lite#7854, should not be super complicated to start with, but I agree, this is upstream work, and always it had been a nice fence, your side/typescript, our side/python. But I'm sure with a few pointers we can get things moving a bit to unclog these issues as well. And without getting anyone involved, maybe @ChristopherDavisUCI is also open to give this a try? What do you think @binste?

[ChristopherDavisUCI]

That sounds very good to me @mattijn, thanks! I got a Vega-Lite development environment working once before (maybe a year ago) and will start out trying to get back to that stage.

[binste]

Sure, totally understand that! And I like the idea of building up the skills to contribute some fixes in the Vega-lite repo ourselves and tear down that fence, I'm in :) I have only some minor javascript experience but will try to brush it up over the coming days and then join the discussion on vega/vega-lite#7854.

[joelostblom]

I think a summary of the outstanding VL issues that a new Altair release is severely affected by would be great and I edited my original post above to include those mentioned here #2702 (comment) (let me know if there are others). I believe it could be helpful to dedicate some effort toward fixing these issues in the Vega-Lite repo, but that we should make sure to prioritize things in the Altair repo still. I am personally unsure about contributing directly to VL at this point in time because I think most of my time will be spent being stuck on simple JS things and environment setup... =/

My understanding of the current VL issues is that none of them is so severe that they would block creating a release candidate without this functionality as long as it is clearly stated in the release notes (which we should create btw #2836). Some of them I think could even be included in a point release of Altair, but let me know if you agree/disagree. Other than that, if anyone of you has the time to drop some review comments on any of these PRs: #2568, #2522, #2574, #2835 that would be great because I think those are all something we would want in an RC to get feedback on.

[ChristopherDavisUCI]

Hi @mattijn and @joelostblom, is there anything I can help with here? Thanks for all your work!

[joelostblom]

Yup I agree with adding the limitations. What do you think of creating an issue when we release the RC and linking to it in the banner of the docs and pinning it to the repo? The issue could read:

Altair 5 release candidate info and feedback

Thanks for trying out the release candidate for Altair 5! If you have any general feedback on the direction of the release candidate, feel free to share it here. If you run into specific problems, please open individual issues for those.

Before you get started, make sure that you read the [release notes] to see what has changed, and that you are aware of the following limitations of the release candidate that we hope will be resolved by the time Altair 5.0 is ready to be released:

• Offline rendering in JupyterLab, using alt.renderers.enable('mimetype'), is currently awaiting an updated renderer in JupyterLab. We expect this to be available soon, in particular, when the PR Switch to VEGALITE5_MIME_TYPE jupyterlab/jupyterlab#13893 appears in a JupyterLab release (likely 4.0).
• Altair 5 will be released with VL5.7, but the 5rc1 is built against VL5.6.1. Meaning that if you are dependent on altair_viewer you cannot update yet, as we plan to release altair_viewer with VL5.7 and not VL5.6.1 (note that altair_saver has altair_viewer as dependency, and that there is a new independent default saving mechanism in Altair 5).
• duplicate parameter definition error in layered charts (because of "point": true) vega-lite#7854 (will be fixed at release with VL 5.7)
• repeat operator with defined width and height does not accept layer vega-lite#8523
• repeat chart with top level defined brush selection corrupts cross-view interaction vega-lite#8348
• chart with repeat and a defined name disables the independency of the repeated channel (eg y-axis) vega-lite#8446

[mattijn]

Also, we should mention that Altair 5 will be released with VL5.7 and that this Altair 5rc1 is build against VL5.6.1. Meaning that if you are dependent on altair_viewer you cannot update yet, as we plan to release altair_viewer with VL5.7 and not VL5.6.1. I think altair_saver has altair_viewer as dependency?

VL issue 7854 is resolved but will be released with VL5.7 so it is still a very important missing feature (maybe the most critical).

[mattijn]

Also, the relevant VL issue/spec is interesting, but I think it’s more relevant to have the Altair spec that one cannot use yet.

[ChristopherDavisUCI]

@joelostblom Do you think it's worth mentioning that offline rendering won't work in JupyterLab until jupyterlab/jupyterlab#13893 appears in a JupyterLab release? I believe this will happen in the next JupyterLab release, but it's possible it will not be until version 4.0. Maybe it could be phrased something like this?

Offline rendering in JupyterLab, using alt.renderers.enable('mimetype'), is currently awaiting an updated renderer in JupyterLab. We expect this to be available soon, in particular, when the PR jupyterlab/jupyterlab#13893 appears in a JupyterLab release.

[joelostblom]

Great ideas! I updated the message above.

[joelostblom]

@mattijn @ChristopherDavisUCI @binste I just noticed this issue in JupyterLab repo jupyterlab/jupyterlab#9929 mentioning that the Vegaj-Lite binaries are fixed at version 4 and it is not looking like anyone is working on updating them for the time being. Would this cause issues for the Altair renderers that render offline? And would this include any common renderers like the jupyterlab one maybe?

[binste]

Not fully sure but isn't this related to #2585? Also the comment by Jake which @ChristopherDavisUCI mentions is probably this one #2797 (comment). So basically a frontend needs to upgrade to vega-lite 5 so that the mimetype renderer and therefore offline rendering works and there is nothing on the Altair side we could do (except for creating a PR in the JupyterLab repo).

[ChristopherDavisUCI]

Thanks @binste, that's exactly the comment I meant. Thanks also for pointing out #2585, that does seem very relevant.

[ChristopherDavisUCI]

Is it clear what would need to be done to update things on the JupyterLab side? I'm happy to try working on that. jupyterlab/jupyterlab#9929 (comment) makes it sound like a PR from us would be welcome.

[joelostblom]

@ChristopherDavisUCI I think you saw it already, but just in case, this post links older PRs doing the same update from dmoritz jupyterlab/jupyterlab#9929 (comment)

[ChristopherDavisUCI]

Thanks @joelostblom, yes, I saw it! I will take a look soon. If it seems too far out of my comfort zone, I'll also let you know.

[jbloom]

@joelostblom, I'm super excited to see all the activity on the GitHub repo from you and others. As an avid user of altair I'm just very curious: what do you expect as likely timeframe to next release?

[joelostblom]

I'm happy to hear you are enjoying all the progress made here lately @jbloom, so am I! We want to have a release candidate "soon", but we're currently waiting to see if some of the upstream issues in Vega-Lite can be fixed before we proceed and publish a release candidate. You can view all the unchecked issues in the original post above to get a rough sense of how close we are to a release / release candidate.

[mattijn]

What would you think of doing a feature freeze at February 26 and around March 1 release of v5.0.0rc1?

[ChristopherDavisUCI]

Sounds exciting to me @mattijn!

[joelostblom]

Those dates sound good to me. I won't be able to work much March 6-14, so releasing before then is desirable. From my end, I would like to see a resolution to #2568, #2824, #2892, and #2871 before the first RC

[binste]

Awesome! 🎉 I created a few more mostly smaller PRs today which would be great to get merged. Open to review other PRs or work on open issues to get everything in until February 26, just ping me.

[ChristopherDavisUCI]

Has anyone checked out the new pandas release candidate https://github.com/pandas-dev/pandas/releases/tag/v2.0.0rc0 to see if it impacts us at all? The jump from v1 to v2 is exciting and a little scary-sounding. (It's always possible I misunderstand, and maybe the upcoming release is always labeled this way?)

[mattijn]

I've a branch where I tested the following Github Actions matrix run succesfully:

matrix:
  python-version: ["3.8", "3.9", "3.10", "3.11"]
  jsonschema-version: ["4.0", "4.5", "4.17"]
  pandas-version: ["1.5.3", "2.0.0rc0"]

But when on jsonschema<4 (eg 3.0.0 or 3.2.0), there is still one test failing, see this action run, same as #2894, but than as error instead of warning.

And using python 3.7 there I can use only pandas 1.3.7, but in combination with jsonschema 4.17 the tests are passing, but there is a deprecation warning (same as #2895)

[binste]

That error is independent of pandas. The test can only ever pass for jsonschema >= 4. I propose a fix in #2931 so that it should work for all jsonschema versions >= 3.

[binste]

Regarding the deprecation warning, I thought I fixed that in #2895 because Python deprecated that interface in Python 3.10. But as Altair relies on the package importlib_metadata for Python < 3.8 (so only 3.7 as 3.6 is not supported by Altair), it's not fixed for Python 3.7. Reason is that the importlib_metadata package also deprecated the interface and this is not captured by the current logic in Altair. sqlalchemy also struggled with this. I'll see how they solved it and will create a PR later today.

[binste]

See #2932

[mattijn]

I just released a new version of Altair (5.0.0rc1) and updated the website at https://altair-viz.github.io/. You can find the package on PyPI at https://pypi.org/project/altair/5.0.0rc1. I'm not sure if we need to release a candidate on Conda yet. Thank you all!

[joelostblom]

Yes, there haven't been much feedback, I guess most things are still working the same as in v4 and we haven't broken anything major which is positive =) It is hard to say how many people have tried out the RC, according to these stats there are around 10k downloads, but we don't know how many of those are part of GitHub workflows etc (although I would expect that share to be lower for an RC).

I am still leaning towards that it makes sense to have an RC2, mainly because it would be nice to release it against VL 5.7 when that is out and at least try it out for a while ourselves and see that things work as expected. I wonder if we should also push the new methods syntax more aggressively in the docs to make it more discoverable and potentially get more feedback from others? In my mind it makes sense to have it as the default in the docs at some point since it its more convenient. Other than that I think we have all the necessary PRs merged already and some that would be nice to have could also be in a minor release.

[binste]

Honestly, I was expecting more issues as well but also my own experience has been very smooth with it so far.

I'd also see VL 5.7 as the only change for which it might be useful to have another RC. Even if it's just out for 1-2 weeks so we can test it ourselves as @joelostblom mentioned. We could also use an RC2 to test and update altair_viewer and maybe altair_saver. Other then that I'm also not aware of any PRs which still need to be merged before a v5 release.

[mattijn]

OK, so lets have another RC2 when we have aligned with VL5.7 when released.
After using the methods-based syntax, I'm also fine with adopting it as default. It is indeed more convenient and a major release can be a good moment to do this.
Also agree that this is also a good moment to see if we can get altair_viewer and altair_saver up and running again with altair v5. I haven't spend much time in these codebases, so I'm a bit hesitant to start, but we can try at least.

[binste]

All changes which are relevant for rc2 are in master as far as I know. No stress at all! Just wanted to ask if there is anything outstanding where I can be of help? I think 1-2 weeks would still be a good length for testing rc2 and then I'm very excited for the official release 🥳 What do you think?

[mattijn]

Let's try to do a rc2-release this weekend and if nothing pops up, aim for an official release in the week of 8 may (before jupycon).

[mattijn]

I just released the second release candidate version of Altair (5.0.0rc2) and updated the website at https://altair-viz.github.io/. You can find the package on PyPI at https://pypi.org/project/altair/5.0.0rc2. to install: pip install altair==5.0.0rc2

[binste]

It uses 5.7.1. In the spirit of a release candidate, I'd suggest that we stick to that version for the official release in case people have tested against it. And then we could do an Altair 5.1 release soon incl. support for the new VL 5.8 PPI feature. Thoughts?

[jonmmease]

I'd lean slightly against releasing the final version with Vega-Lite 5.7 because I'm not clear on how the issue in vega/vega-lite#8841 (which is fixed in 5.8.0) might impact Altair users.

[mattijn]

I tend to agree as well, but I like to have a confirmation first, since the test suite depends on altair_viewer which includes the VL and Vega versions of the actual releases of Altair.

[mattijn]

But if we release against vl5.8 we also should cut a rc3 release soon if we want to release next week an official release.

[mattijn]

I've prepared altair-viz/altair_viewer#57 and #3037. Once we have confirmation if vega-lite 5.8.0 depends on vega 5.25.0 (@kanitw, @domoritz). I can start working towards releasing rc3.

[mattijn]

I just released the third release candidate version of Altair (5.0.0rc3) and updated the website at https://altair-viz.github.io/. You can find the package on PyPI at https://pypi.org/project/altair/5.0.0rc3. to install: pip install altair==5.0.0rc3.

I had some connectivity issues during uploading to PyPI, but everything seems to be uploaded fine.

[mattijn]

I will start making the v5.0.0 release. Hopefully done within one hour

[binste]

This is very exciting 🥳 Thank you @mattijn!