Automated tests on examples lifecycle (template init, template create, workspace create, ssh)

[bpmct]

What is your suggestion?

Run (nightly?) automated tests for all example templates to identify breaking changes or regressions.

I believe this would have helped identify:

• fix: Use environment variables for agent authentication #1238
• Bug: Obscure error when using Docker template #1198
• Bug: Template parse with fails EOF, parse source: session shutdown #1381

as well as any flakes in the examples.

Why do you want this feature?

Ensure examples in the repo are stable and work with the main branch.

Are there any workarounds to get this functionality today?

Manually testing each example each release.

Are you interested in submitting a PR for this?

No

[tjcran]

@misskniss @bpmct
If this can't be automated, it's something we'll need to do as a process thing on a regular cadence for any templates we have as "official" examples in the repo. I don't think we'll necessarily need to do this weekly right now, but we should do a team bug bash right before we "flip the switch" to test each example and then again before we do the public launch. How does that sound?

[tjcran]

I don't think we can add the automation for this to the Community launch. But should be investigated for the next milestone.

[bpmct]

Going to reopen - we've had a couple of requests for this, specifically around if dev workflows work from version-to-version (python3 command works, web app runs as expected, etc)

[github-actions]

This issue is becoming stale. In order to keep the tracker readable and actionable, I'm going close to this issue in 7 days if there isn't more activity.