runner-images experience feedbacks.

[AlenaSviridenko]

Hi there!

One of our goals as a maintainers of this repo is to provide nice and smooth customer experience with virtual-environments.
We would like to open this discussion to hear your feedbacks on service usage and how we could improve your experience.

Please, share here your thoughts\ideas about potential process and repository improvements 📈

[asklar]

• Whenever new VS updates get pushed (or new tool versions are adopted), it usually breaks our project in some way, causing lots of disruptions for ourselves and our customers. Discoverability of upcoming breaking changes is still difficult, so I've opted for watching the repo. I get tons of mail about it, most of it is unrelated/irrelevant for my purposes (mac/ubuntu), but our team has been burned by not knowing about breaking changes multiple times already. Watching releases only doesn't seem to work as in the past I believe releases would get pushed post-hoc after an update, by that time we've already gotten broken.

• on the same vein, being able to opt-out of breaking changes until we have a chance to update our code, would be good. Often virtual-environment takes VS updates as soon as they're available, which doesn't give us enough time to try the VS update.

• Might be unrelated to virtual-environments but still relevant: There have now been at least 2 azure incidents that have taken down our CIs due to regressions - not in the images necessarily, but in tasks like publishing artifacts. There was one regression that was introduced because it didn't take into account artifacts bigger than 2 GB (which we have some). Increased testing of changes, please.

[maxim-lobanov]

Thank you for feedback @asklar.

• Unfortunately, we experience the same problems and frustrating about Visual Studio updates. Usually, release notes don't describe breaking changes in Visual Studio and some updates (for example, 16.8 that brings .NET 5) could be very painful.
  The other problem is that Visual Studio doesn't provide way to install non-latest version or freeze some specific version (At least, we don't face it). Even if we notice some breaking changes or bugs in Visual Studio, we are not able to prevent picking up them on image-generation.
  We can't stop updating images because it is not acceptable for our case and any new image, that is generated after release of new VS version, will automatically pick it up.
• Partially answered above. We are looking at some ways how it can be improved and ability to provide previous image for some more time. If the latest image update / VS release breaks you, you might temporary switch to previous version of image to unblock important releases. But there are still some technical limitations around it and there is no ETA or clear solution here.
• Sorry, I can't provide much context here. I think it should be reported to https://github.com/microsoft/azure-pipelines-tasks

[asklar]

thanks @maxim-lobanov . I'd be very interested in the "revert to a previous image for a time" feature. If/when there is a way we can try it out internally at msft please let me know :=)

[dentuzhik]

While this thread is about VS updates, there are similar issues coming from another sides:
#2420 (comment)

It would be nice if we would have a better channel for announcing potential backwards incompatible updates.
A good improvement would be to use at least Github Changelog, however I must say there are still now ways to actually "subscribe" to it.

on the same vein, being able to opt-out of breaking changes until we have a chance to update our code, would be good. Often virtual-environment takes VS updates as soon as they're available, which doesn't give us enough time to try the VS update.

💯 It would be ideal to eliminate possibility to break CI by third-party by giving consumer an option to specify the version of virtual environment his workflow is running on. While this is not the case of 99% of the workflows we write, for some it would save effort debugging later and having releases blocked.

[Chireven]

Greetings, and thank you so much for the feedback.

I'm coming at this with no experience in a lot of the software used in this project. I'm not a developer, I don't use most of the programs in it and I've never used packer before. With that said, I have been struggling to make this just spit out an image as it keeps failing. The biggest problem is that it fails after 2-3 hours of ... doing something,

Our developers are using Azure DevOps. We have come to a point where the default microsoft build agents aren't cutting it, and I'm evaluating bring them on prem or hosting them as a Scale Set in Azure. Regardless, I need to spit out a template to use as a baseline. I was excited to find this project last week as building a template using the same logic Microsoft does should get me exactly what our developers need.

I've run into a few problems:

1. The learning curve is steep. I understand what packer does, but it seems overly complicated for what its doing. It's a very interesting tool and I'm already looking to adopt it in other areas. I think the worst part about it is honestly the name -- when searching for information, you most certainly have to filter out a lot of football results!!!!

2. Success is dependent on third parties. I lost half a day trying to figure out why something was failing, and it just started working. Upon investigation, it turns out a site was having problems that the script was trying to pull files from.

3. It takes way too long - at least for building Windows machines. If I understand things right, It seems to be related to WinRM. I spent quite a bit of time adapting the template to use SSH and ran into even more problems. For example, things were failing with long filenames (or possibly file./folder names with spaces). It just needs to be faster!

4. I'd like to see some more switches to use dynamic builds. For example, everything assumes you're going to use Azure -- and I probably will. However, I'd like to build one local to get a feel for everything -- and keep it local for troubleshooting. It would be nice to be able to add a -builder parameter and pass it builder type and the scripts just take care of all the differences between builders.

5. There is no way to skip a failure -- its just restart or clean up and abort. Right now I'm trying to figure out why docker is having problems (service isn't starting on boot). I don't even need it in the image, but I can't move past it.

6. A little bit of guidance on the main page would go a long way. It took me quite some time (and I consider myself to be technical) to figure out how to get started.

7. It probably goes against what you're trying to do here, but I'd love to be able to just run something in a virtual machine and it puts down all the software.

[jmos5156]

Hello.

It has been a while since I last used this repo but now find myself drawn back in due to further business requirements. For the record, I think this is a great repo, and love the work that's being carried out. I have been using images created from a fork of the repo for quite some time and without too many issues.

I refer you to my original git issue I posted last year #706 as I used this to pick up from where I left off last year and indeed see a lot of great changes including the use of Pester testing for better reporting and troubleshooting of the image creation process. Nice job!

Coming back to this repo I still find it hard to get up to speed and quickly start using this great work to build my own images. I understand that your horizons might be different from mine but I feel that they are nevertheless grounded in much the same code. Build images from a library of available software. Whereas you want to build it ALL, I want to build smaller faster subsets of what is on offer to suit my business needs. However, I do not want to diverge so much that I spend days remerging new work and trying to refix, relearn or redesign to take advantage of what new features you have to offer.

Much of this has been aligned into a due process, such as

• separating data from code in the form of toolset JSON files and consolidating that away from the Packer manifest and Install scripts.
• Using Pester testing for post-install and final validation

However, I feel more could be done. To that end, I will list my ideas below with as much detail as possible. With any of these, I am open to discussion, and if you feel worthy happy to help contribute with pull requests.

All of the suggestions below stem from one ideal, that I should be able to pick and choose what software I want to install with the minimal amount of change, yet remain as closely aligned to the repository as possible.

1. 7Zip should be mandatory. Let's face it, it's defined as a function in InstalleHelpers.ps1 and used throughout the install scripts (CloudFoundry, Chrome, FireFox, PyPy...). If it's that important just install it. Add it to the Initialise-vm.ps1 script and have it installed straight after Chocolatey is installed. One less script to worry about.

   	Invoke-Expression ((new-object net.webclient).DownloadString('https://chocolatey.org/install.ps1'))
   	Choco-Install -PackageName 7zip.install
   

• Other dependencies should be installed as a matter of course

   	Choco-Install -PackageName vcredist-all #Visual C++ Redistributable Runtimes
  

1. All installs should be complete. It should be possible to install each independently of each other

   • If TypeScript needs NPM it should test for its presence. If there used it. Otherwise use Chocolatey

   	if ( (& npm -v)){
   		npm install -g typescript
   	} else {
   		choco install typescript
   	}
   

2. Installs should be grouped together by type and common env vars in provisioners (where possible). If they must have dependencies between them then these should be placed together for quick understanding if documentation is lacking. There are numerous places where there are dependencies between unrelated software.

   • Finalize-vm has a dependency on yarn which is installed in the NodeLTS script. If I remove one the other fails. Safer to add a yarn test in the Finalise script to avoid errors

   	if ( (& npm -v)){
   		# Clean yarn and npm cache
   		yarn cache clean
   		npm cache clean --force
   	}
   

   • Install-TypeScript has a dependency on yarn also
   • Install-PyPy requires the x64 C++ Redistributable and the Install-Ruby.ps1. I have found that the Initialise-VM script installs the x86 version as part of Chocolatey install. I also assume that VS studio also installs x64 version too. This gives a successful build. However, when installed separately it fails.

3. Break out installs and pester tests as individual steps.

   • Helm is installed alongside Docker in the Install-Docker.ps1. This seems unrelated to me and I would break this out into an Install-KubernetesHelm.ps1 or at least move it into Install-KubernetesCLI.ps1. The same is true of the pester tests.
   • MiniKube is installed as part of Kubernetes CLI. I can see the relationship but I think it best to define that in a separate Install-KubernetesMiniKube.ps1 script.
   • The above example could be added to a packer provisioner that deals with Kubernetes install

   	{
   		#Kubernetes installs only
           "type": "powershell",
           "environment_vars": [
               "TOOLSET_JSON_PATH={{user `toolset_json_path`}}"
           ],
           "scripts": [
               "{{ template_dir }}/scripts/Installers/Install-KubernetesCli.ps1",
   			"{{ template_dir }}/scripts/Installers/Install-KubernetesHelm.ps1",
   			"{{ template_dir }}/scripts/Installers/Install-KubernetesMiniKube.ps1"
           ]
       },
   

4. Add all env vars in one provisioner. Currently, we define these per provisioner and in cases is repeated. Add it once at the beginning and should any script need it, it is already there. Maybe this could avoid issues further down the line?

   {
   	"type": "powershell",
   	"environment_vars": [
   		"IMAGE_VERSION={{user `image_version`}}",
   		"IMAGEDATA_FILE={{user `imagedata_file`}}",
   		"TOOLSET_JSON_PATH={{user `toolset_json_path`}}",
   		"ROOT_FOLDER={{user `root_folder`}}",
   		"PSMODULES_ROOT_FOLDER={{user `psmodules_root_folder`}}"
   	]
   }
   

For the moment this is what I have come across in remerging from your repo. Should I come across anything else or have ideas I will post here first.

[AlenaSviridenko]

@jmos5156 thank you for the write-up, it all makes sense to us. We will create several work items based on it and implement when we have capacity.

[speedmind]

Hi,

I've been referring to the installed software portion of this repo for various images being used in Microsoft Hosted Agents for quite a while (also the older vsts-agent-docker repo).

For various reasons (VPN, etc) I need to be able to run the same agents on-premise. I'm considering building the images I require from this repo, but was wandering if they exist as a docker image (I'm going to be running in an AKS cluster) in a public registry somewhere.

So far I've not seen any documentation (despite spending quite a bit of time searching for it) and I've been through the scripts to see if there's anything related to tagging but no luck so far.

Have you considered publishing these images to a public registry at all for consumption by "docker/k8s build agents"?

[catthehacker]

It was already discussed here: #2320 (comment)
It's also not possible to redistribute this images due to licencing issues

[speedmind]

Thanks a lot for the prompt response, much appreciated.

[joto]

I am developing on several Open Source repositories and checking whether the software still runs in multiple environments is essential. I want to make sure the software runs on all systems that users are likely to have. This includes current as well as older systems.

The biggest problem with Github Actions environments I have is that, more often then not, breaking builds are not due to some problem in my code but due to changes in the environment. I really need stable environments of popular stock systems in several versions. Most recently some older versions of the clang compiler were removed which failed the build. But I am using those old versions to make sure the users of those old systems can compile my program with out-of-the-box tools that come with that system. It is okay to have some system that is updated all the time to get the newest and fanciest, but I also need systems that will not change (except for things like security updates to the base system) so that I can be sure that if something is red, it is my fault.

[AlenaSviridenko]

Hi @joto, thank you for your feedback!

You caught it right, our main idea is to provide the most recent software on our images, and yeah, unfortunately, sometimes we have to deprecate some old software to release space and simplify maintenance of the images. We always try to announce software removal in the image's README files in the announcements section at least 1 week prior to changes. You can subscribe to our releases as they contain announcements too https://github.com/actions/virtual-environments/releases.

We have an idea on our radar to provide at least one previous image in order to give an ability to switch workflows back to this version and give some time to adapt to the changes, but we cannot afford to keep a lot of old images because it will complicate the image and infrastructure maintenance process significantly.

[joto]

Keeping up with changes week by week is impossible for, I would guess, almost everybody. And even if I could, as I said, it is important for me to test on a variety of systems. That's the unique thing that a CI system offers above what I can test on my own system. Having the newest software on the CI system is nice, but it is only a small part of what the CI system needs to provide. To make sure my software runs on all systems I need a stock Ubuntu 18.04 and stock Ubuntu 20.04 and a stock Redhat and a stock OSX whatever etc. The vendors already are doing the hard work of releasing systems and keeping them up-to-date. You can leverage their work and just provide the images based on what they offer. There isn't really any maintainance needed except updating those systems whenever the vendor releases updates or new versions.

[catthehacker]

If you require plain OS you can use container: option in workflows or use self-hosted runners.

[dentuzhik]

Hey guys!

I have drafted another discussion thread here, which is related to the previous thread
#3198

[xavier2k6]

In relation to supporting at least 2 latest OS versions (LTS only for Ubuntu)

Currently the OS Specific-latest label is used for the most recent OS
Example:
Ubuntu-latest = 20.04

Recently macOS-11 was made generally available ref.: https://github.blog/changelog/2021-08-16-github-actions-macos-11-big-sur-is-generally-available-on-github-hosted-runners/

Yet this hasn't been given the macOS-latest labelling?!

I can only assume that the same will happen when windows-2022 comes out of "beta"....

How long before these released/newest OS's get labelled as xxxx-latest?

[maxim-lobanov]

Hello @xavier2k6 , usually it takes ~6 months.

Since macOS-11 was in private beta, the number of users and feedback was limited.
After GA, users has started to switch to the new OS. Before migrating -latest, we need to make sure that there is no any issues with the new image.
Another point is proper communication. We need to announce migrating latest tag beforehand because switching tag is always painful for users.

[xavier2k6]

@maxim-lobanov thanks!

[joto]

I don't understand your policy regarding upgrades. The whole point of having a CI is that you have a stable environment, so if anything breaks, you know your own code broke and you have to fix it. But my experience is the opposite: Most of the time if something breaks, it is not my code, Github changed the environment the code runs in. This totally defeats the purpose of having a CI in the first place!

[henryiii]

I disagree. If you care about specific versions, you should use docker and/or pin those versions manually - there are lots of actions that help with that, etc. If you don't care, you should get the latest versions. If upgrading to the latest versions breaks your code, your users will break too. I don't think any CI system does "stock" os's, that's what containers are for.

[donk-msft]

My goal is to automate the creation of an Agent image. Use it to deploy a scale-set and add the scale set to an Azure DevOps Agent Pool.

I tried to follow the instructions in the 'create-image-and-azure-resources.md' document yesterday and my initial feedback is:

• The guidance seems to be written for people who live and breathe this stuff. It feels a bit overwhelming to get started if you're new to the topic.
• My vanilla Azure Windows 2019 VM doesn't have chocolately or git installed. Installation of these prereqs for example is not in the documentation.
• I'd be great if there's are template pipelines available that automate the image generation, VMSS deployment and adding to the ADO agent pool steps.

[miketimofeev]

Hi @GitHubUser1339!
Please feel free to contribute to the docs if you see that some parts are not covered. We don't have experience with VMSS deployment thus can't help much with the script.

[donk-msft]

Automation of the image generation part would be really helpful, as it's the most complicated part.

[donk-msft]

I found a very useful solution here

[ndinev]

I am just wondering is there a way to have aditional toolset(s) defined for a minimal installation. Curretnly the install and config of Windows 2019 takes 5-6 hours. If we have some minimal installation it will be easier to understand how to gradually add commponents needed. Just imagine that you want to run java selenium tests - so you will need only java, selenium, cromium driver. It will be easier to add them to minimal install instead to try to remove almost everithing from full install

[lzandman]

This week I spent some time generating custom runner images and I have a few observations. I'm still in the process of grasping all of this, so forgive me if some of my observations don't make any sense...

1. Image generation is too slow, especially for Windows runners (more than five hours for the Windows 2019 image!). I've seen there's already a topic on this. I would be great to see some improvement here.

2. My strategy for creating my own custom images was to take one of the existing HCL2 templates and just comment out the things I thought I didn't need. I was surprised to see this was actually quite difficult, apparently because just taking out an Install-* script does not seem to always remove all accompanying test cases. While each install file may also run its tests, it appears additional tests are run at the very end of the image generation.

   Also, while some applications are installed using a clearly named Install-* script (i.e. Install-Apache.ps1 installs Apache), other applications get installed from scripts that install multiple applications (i.e Install-ChocolateyPackages.ps1, which installs multiple packages using Chocolatey, each defined in a toolset file). Things would be a lot easier/clearer if every application would have its own dedicated install and test script.

3. There's a MinimalUbuntu HCL2 template, but no minimal template for Windows and Mac. Why? I get this project is mainly intended for building Github/Microsoft's own runners, which are very fat, full featured images. However, my guess is that most organisations who wish to use this project as a basis for their own runner images don't want/need such fully packed runners. It would make sense to offer minimal runner templates for each supported runner OS. Such a minimal runner template would contain code to provision only the OS and the minimally required runner tools/helper framework scripts.

   It would then make sense to base the other, more full-featured, runner HCL2 templates on these minimal templates. I don't know enough about Packer yet, but wouldn't it be possible to have a minimal HCL2 template and have the extended HCL2 templates include/refer to that base HCL2 file? I.e. split up the HCL2 template files into multiple files and make it more modular?

4. The install helper scripts have a dependency on certain tools. For instance, for my custom Windows image I thought I didn't need Install-ChocolateyPackages.ps1 script, so I had commented out that line in the HCL2 file. However, during image generation some errors occurred. It appeared Install-JavaTools.ps1 uses the Expand-7ZipArchive() helper function, which in turn needs 7-Zip to be installed. And 7-Zip gets installed from the Install-ChocolateyPackages.ps1 script. So these two scripts have a dependency, which I think isn't obvious.

   I think it would be helpful to more clearly define/document the base runner image codebase. Clearly mark what is minimally needed to have a fully funtional base runner, including a fully working helper function library. I.e. the things that have to be installed and available, before other applications can be added using the helper functions.

   For the above mentioned minimal templates, this would mean such a minimal template would also have to include this base set of applications/scripts required.

5. The image generation code uses the concept of a "toolset" file, a JSON file that contains a collection of data structures that are used during deployment. I get the feeling this file initially had a clearly defined purpose, but over time got "abused" by adding more and more functionally unrelated sections. There's data for installing multiple versions of an application. But there's also data for installing multiple NPM and Chocolatey packages. I find this whole concept rather confusing. Why does 7-Zip get installed using that toolset file? Why isn't there a dedicated Install-7Zip.ps1 script, which in turn could use Chocolatey to do its work?

   Also, there are several Install-*.ps1 scripts that make use of the data inside the toolset file. Why not split the toolset file into multiple multiple toolset files, one for each specific section, and actually specify the used toolset file in the command-line? So, instead of calling Install-ChocolateyPackages.ps1 you would call Install-ChocolateyPackages.ps1 -ToolsetFile "chocolatey-packages.json". That way it would be immediately clear which file is used and for what purpose.

6. When you run the GenerateResourcesAndImage script, it only logs to the standard out. I get this is the preferred way when calling it from a CI/CD environment, where standard-out logging is typically augmented with datetime stamps and stored. However, when running this script stand-alone from a terminal it would be nice if the logging was augmented with timestamps and also, optionally, gets written to a file.

7. The CreateAzureVMFromPackerTemplate script has a ResourceGroupName argument that is used both for the location of the runner VM image and the location where the actual VM instance is created. Wouldn't it make more sense to create the VM instance in a separate resource group? That way you can easily delete that whole resource group, without risking that you delete the precious VM image itself (which, as said above, might have taken hours to create).

8. The password for the temporary Packer VM is generated at runtime, based on the username and a fresh new GUID, but isn't stored anywhere. Of course this is safe, but it also makes debugging a lot harder. It would be nice if that password gets logged or maybe stored in a key vault, so it is retrievable.

9. The ubuntu-minimal.pkr.hcl HCL2 file contains comments explaining some of its contents. The other HCL2 files don't contain such comments. It would be nice to clearly document sections of all HCL2 files using comments.

[m-soltani]

I just wanted to reach out and say thank you for your consistent work on this product. We in our company really appreciate your work here, with a little bit of modification (like publishing images in an azure compute gallery instead of VHD images), we now have all our self-hosted build servers always up-to-date and we can roll out tens of build servers at scale, without any interruptions). Thank you 👍🏻

[rybal06]

Likewise! The amount of effort that goes into maintaining a solution like this is huge. We could not do a better job ourselves and this makes us incredibly more productive.

It took a lot of time to get this incorporated into our own orchestration for managing image versions and deploying new versions as rolling updates; but being able to host near the exact same solution as MSFT within a private virtual network (as MS-hosted agents do not support private network integration) has been a huge win for my organization. We are confident that all tasks which run on MS agents will run on our agents. The ephemeral model with VMSS means we can safely share agents across workloads. That in itself is a massive reduction of overhead vs. hosting dedicated CICD agents for each workload.

The only thing that would be better is if the MS-hosted agents supported virtual network integration - but the work in this repo would be needed either way. It's great the MS took the effort to open source this repo so customers like us can utilize the same solutions internally to workaround the private network limitation of the MS-hosted agents.

[m-soltani]

I agree with you but there's a significant need for improvements in the area of Virtual Machine Scale Sets (VMSS). The fact that it can take up to half an hour to spin up an instance and register the agent for a pipeline to run presents a considerable challenge for our adoption of VMSS. Additionally, the limitations on parallel jobs and the need to pre-purchase capacities are further obstacles. Regardless of where you host your agent, whether it's in VMSS or a Kubernetes cluster, you're still limited by the number of parallel jobs your organization has configured and paid for. We haven't yet explored competitive solutions and their hosting models, such as GitHub Actions, but overall, we are quite satisfied with our current build infrastructure.

But yeah, always room for improvements on the infrastructure side.