[Poll]: Experimental versions of dxvk, vkd3d

[cyberphantom52]

As you all know, Proton Experimental as well as ProtonGE both ship the bleeding edge git versions of dxvk and vkd3d, which benefit from latest developments regarding compatibility for new game releases as well as new features. These versions being bleeding edge also have the downsides of potentially breaking games.

Considering all this a poll is being created to decide whether those versions should be available for installing directly from within Bottles for people who want test newer versions.

Those versions won't be downloaded by default and users who want to try them have to do so by manually installing them from components page.

Please upvote or downvote depending on your preference.

[arcDaniel84]

having the choice is always great so I vote for it an ask for an other option: to chose the proton versions from the installed steam (heroic can do that)

[keenanweaver]

Might be outside the scope, but if this gets approved, then it might be beneficial to create a CI to pull the latest once a day or something like that, alongside custom Wine runners (Lutris, WineGE, etc.), and DXVK-NVAPI/LatencyFlex. Should be pretty easy to set up in Github Actions.

[koplo199]

@keenanweaver Yes, by chance both DXVK and VKD3D-Proton already have their CI set up with Github Actions, so it is only a matter of using this action with workflow_conclusion: success and check_artifacts: true to pull artifacts of their last successful build and tada : no need to set up an environment and do custom builds at all!

Concerning other components/runners I would say that WineGE is already released fairly regularly, and DXVK-NVAPI/LatencyFlex are nowhere near as critical as DXVK/VKD3D-Proton but hey, they also both have their CI set up with Github Actions so it wouldn't cost anything to add them too.

In fact the most complicated might be UX wise : after choosing to install let's say dxvk-master-0462454 in the menu : what is the behavior? Does it get automatically updated to a newer build at startup or does it require a manual intervention? If automatically updated, what if the new version contains bugs, is that possible to revert to the previous build? If so, how many builds does it need to keep in cache locally, all of them until manual erasure? etc.

Imho it could be a very confusing/frustrating feature if there is some sort of automation involved (in addition to be a lot more painful to implement and prone to failure), so it may be better to let the installation be manual at all times : keep showing the currently installed version (e.g dxvk-master-0462454), and show one (and only one) eventual newer build above it if it exists (e.g dxvk-master-0b9f785), letting the user installing it if he wants to. Then, if the user decide to update, keep the 'old' dxvk-master-0462454 build until manual deletion, and never show it again after that. Could it be an acceptable workflow?

[keenanweaver]

so it really is only a matter of using this action with workflow_conclusion: success and check_artifacts: true to pull artifacts of their last successful build and tada : no need to set up an environment and do custom builds at all!

Since Bottles works with manifests, it'd be retrieving all the information regarding an artifact or release and generating the manifest based on that info, then, of course, automerging to the repo. Not sure how it works on the backend if the changes are immedately shown to the user once a change is merged, but from my PRs to the components repo, it's been fairly fast.

The pieces of information we'd need to get from each component are the following:

• Name of the component
• Provider of the component
• Channel (will be 'experimental' or whatever it'd be called)
• Filename
• Download URL
• File checksum (md5)
• Renamed file (optional)

Examples:

Name: dxvk-async-1.10.3
Provider: Sporif
Channel: stable
File:
- file_name: dxvk-async-1.10.3.tar.gz
  url: https://github.com/Sporif/dxvk-async/releases/download/1.10.3/dxvk-async-1.10.3.tar.gz
  file_checksum: 7132c23fe8d7b1353509e26d76bf153c
  rename: dxvk-async-1.10.3.tar.gz

Name: wine-ge-7-29
Provider: gloriouseggroll
Channel: stable
File:
- file_name: wine-lutris-GE-Proton7-29-x86_64.tar.xz
  url: https://github.com/GloriousEggroll/wine-ge-custom/releases/download/GE-Proton7-29/wine-lutris-GE-Proton7-29-x86_64.tar.xz
  file_checksum: 077a5548a533e17f9d1c9f8a0e9dcd02
  rename: wine-ge-7-29.tar.xz
Post:
- action: rename
  source: lutris-GE-Proton7-29
  destination: wine-ge-7-29

All of the above should be easily attainable using jobs or the Github API. Using the script here might be good enough for the CI. But it's probably better to translate as much as possible to Github Action language instead of running scripts.

In fact the most complicated might be UX wise : after choosing to install let's say dxvk-master-0462454 in the menu : what is the behavior? Does it get automatically updated to a newer build at startup or does it require a manual intervention? If automatically updated, what if the new version contains bugs, is that possible to revert to the previous build? If so, how many builds does it need to keep in cache locally, all of them until manual erasure? etc.

I could see it going several ways. There could be a generic 'dxvk latest experimental' or something entry, which manages it all for the user. Or it'd be how it currently works where it's just a list of available components to install. I don't think Bottles should try to auto-update or manage stuff for the user unless it was super, super stable.

What should be automated is the detection and merging of components to the repo so the changes appear to the user quickly without waiting for some person to submit a PR and get it approved. Someone out there really wants to run the latest WineGE but might have to wait a few days if a PR isn't submitted & approved.

Agreed regarding the build list. It'd just be a giant list of experimental builds for each component deemed worthy of tracking artifacts. Maybe have it clear out once a month and retain the ones the user has installed in the list. All I know is it will be a bad experience having to scroll through hundreds of artifacts in the Bottles UI.

[koplo199]

Since Bottles works with manifests, it'd be retrieving all the information regarding an artifact or release and generating the manifest based on that info, then, of course, automerging to the repo.

Unfortunately, the CI would need to do more than retrieving information about artifacts, it needs to download them for at least 3 reasons :

• A Github account/API key is needed to retrieve the artifacts.
• Workflows are temporary and could be deleted at any time, worst case scenario the URL described on the manifest info isn't valid anymore.
• Artifact archives would need some modifications : DXVK for instance include native libs in them, which are not needed by Bottles, and represent 50% of the archive size (~20 mb). It could add up pretty quickly if having multiple version of each components.
  (Also archives are published as zip and not as tar as needed, and the retrieved hash would be SHA1 instead of MD5, requiring some change to the backend.)

This is why I linked an artifact downloader action earlier, I guess the simplest effective CI would be looking roughly like the following :

1. Triggered on a time basis (every day/week)
2. Download all artifacts of chosen repository
3. Do post-processing on archives (removing unnecessary files, compressing to tar.gz instead of zip, etc)
4. Generate manifest entry accordingly
5. Publish manifests and newly created artifacts as release

Using the script here might be good enough for the CI. But it's probably better to translate as much as possible to Github Action language instead of running scripts.

I don't know, python could be installed on a runner in one line, and maintaining one script look easier than maintaining two so it may be worth to keep using MaintainersHelpers after all.

I could see it going several ways. There could be a generic 'dxvk latest experimental' or something entry, which manages it all for the user. Or it'd be how it currently works where it's just a list of available components to install. I don't think Bottles should try to auto-update or manage stuff for the user unless it was super, super stable.

For the name, maybe it could retains the same convention introduced by vkd3d-proton-2.6-1-5b73139? Having the commit sha could be interesting if looking for a particular feature introduced or bug fix.

What should be automated is the detection and merging of components to the repo so the changes appear to the user quickly without waiting for some person to submit a PR and get it approved. Someone out there really wants to run the latest WineGE but might have to wait a few days if a PR isn't submitted & approved.

It would be really great, I was so focused on incremental builds that I didn't even think the same process could be done for official release!

Agreed regarding the build list. It'd just be a giant list of experimental builds for each component deemed worthy of tracking artifacts. Maybe have it clear out once a month and retain the ones the user has installed in the list. All I know is it will be a bad experience having to scroll through hundreds of artifacts in the Bottles UI.

Even now, once installed it is forever retained until manually deleted (by clicking on the trash icon in the UI) so we don't need to worry about this part, but you're right the 1 million dollar question is : how many experimental builds is too many experimental builds? 😳 This poll would probably gives some insights.

[koplo199]

Hi,
I got a working prototype which automate the process of updating the components repository, it currently works with stable version of:

• DXVK
• VKD3D-Proton
• Soda
• Caffe
• Wine-GE
• Proton-GE
• LatencyFlex
• DXVK-NVAPI
• DXVK-Async

And experimental version of:

• DXVK
• VKD3D-Proton
• LatencyFlex
• DXVK-NVAPI

An example of pull request created by the CI here. You'll see that the index.yml has been split into multiple yml files in the input_files directory, and is now generated from them automatically.

The only downside is that it requires a major rewrite of the Bottle source code... Just kidding (:sweat_smile:), it is working outside of the box for every listed component, except for Wine-GE which needs a minor change in preferences.py: replacing occurences of wine-ge by wine-GE.

The CI is generic enough so that virtually any component can be added if it could be described in the following format:

{repo : "doitsujin/dxvk", workflow: "artifacts.yml", branch: "master", name-prefix: "dxvk-", version-prefix: "v", Category: "dxvk", Sub-category: "", yaml-file: "14-dxvk.yml" },

The reason why wine-kron4ek for instance isn't currently supported is precisely because it couldn't, as it would require the introduction of at least one supplemental parameter like name-suffix (to support the -amd64 / -staging-amd64 / -staging-tkg-amd64 naming thing), and I don't want to complexify the code any further.

The next steps would be:

1. Write a bit of documentation and add comments to the code.
2. A lot of refactoring/polishing, and contribute upstream for some of the Actions used.

Maintainers, please tell me if it's something worth keeping working on it, just so I don't invest more time in it if it's not. For the record, I currently spent one week worth of work in it.

[mirkobrombin]

Nice job! I would prefer to keep wine-ge in lowercase if possible, for consistency with the other runners (GE-Proton will be renamed too)

[koplo199]

Nice job! I would prefer to keep wine-ge in lowercase if possible, for consistency with the other runners (GE-Proton will be renamed too)

Thanks! Well if GE-Proton will be renamed too then no problem: a toLowerCase() or equivalent will do the job just fine (I just wanted to avoid workarounds for specific components). I will add it in the next revision

[koplo199]

@mirkobrombin I've updated the workflow and it now supports both ge-proton and wine-ge in lowercase 👍 (I'm also done refactoring)

However I've spotted an issue that I didn't see before: it seems that wine-ge is getting a special treatment with

Post:
- action: rename
  source: lutris-GE-Proton7-34
  dest: wine-ge-7-34

In your opinion what would be the most simple way to support it without resorting to wine-ge specific hacks in the workflow?
Here is the YAML that is currently generated by the CI using the MaintainersHelpers/component-generator.py script

[mirkobrombin]

I think we can't escape from this workaround, since wine-ge changed its name months ago from GE-Wine to Wine-GE, plus the project name is Wine GE but the archive is wine-ge-lutris which may confuse users a lot

[koplo199]

Ok, I'll add it in the next revision

[keenanweaver]

Is it in the cards to set up an auto approve action? It's wonderful the PRs are automated, but the cherry on top would be auto-merging. It's probably wise to see the CI in action for a few cycles before setting up auto-approve, though.

[mirkobrombin]

Is it in the cards to set up an auto approve action? It's wonderful the PRs are automated, but the cherry on top would be auto-merging. It's probably wise to see the CI in action for a few cycles before setting up auto-approve, though.

I agree

[koplo199]

Sure, I need to open a PR and get it merged for dawidd6/action-download-artifact#210 (otherwise perfectly valid runs like this won't pass the CI), but after that it should be easy to implement as the action you linked seems really straightforward to use

[koplo199]

Basically

[koplo199]

My PR for dawidd6/action-download-artifact#210 got accepted, so as requested I implemented the auto-merging feature: it uses the native auto-merge GitHub feature (so it will not conflict with any other checks if more are added in the future) in conjunction with the auto-approve action that @keenanweaver linked

It needs:

1. Auto-merge enabled in the repository settings (no shit)
2. Targeted branch configured to require 1 approval (but as stated above, any other checks could be added at any time)
3. A PAT with public_repo scope from any account (required because GitHub won't let anyone approve their own PR, @github-actions being no exception, the PAT is only used to create the PR from another account, hence why only the public_repo scope is required)
4. It is disabled by default, the false in lines 234 and 242 needs to be removed to be activated

[koplo199]

The pull request is now live 🎉