Support experimental features

[bahmutov]

• Closes Allow running new features behind an experiments flag or variable #6257

User facing changelog

Additional details

• boolean flags only for now from process.env.CYPRESS_EXPERIMENTS. Unknown experiments are ignored
• trying it in action for component testing feat: improve component based tests approach #5923
• on the Node side use packages/server/lib/experiments.ts

// example
const experiments = require('./experiments').getExperiments()
if (experiments.componentTesting) {
 ...
}

• on the desktop GUI side, we grab the environment variable EXPERIMENTS from resolved config and parse it using the same logic from packages/server. Which means technically the experiment can be overwritten using plugins file or CLI for the browser and desktop GUI, while only getting value from the environment variable on the Node side

How has the user experience changed?

There is a new tab and by default, since there is nothing whitelisted in the config, it is just notification text

PR Tasks

• have GUI and terminal changes been approved?

Then:

• Have tests been added/updated?
• Has the original issue been tagged with a release in ZenHub?
• Has a PR for user-facing changes been opened in cypress-documentation?
• On links updated and released for https://on.cypress.io/experiments

Plan is to merge this PR with componentTesting or empty list of experiments (?) and then merge #5923 with componentTesting experiment (and a few tests for new mounting)

[cypress-bot]

Thanks for the contribution! Below are some guidelines Cypress uses when doing PR reviews.

• Please write [WIP] in the title of your Pull Request if your PR is not ready for review - someone will review your PR as soon as the [WIP] is removed.
• Please familiarize yourself with the PR Review Checklist and feel free to make updates on your PR based on these guidelines.

PR Review Checklist

If any of the following requirements can't be met, leave a comment in the review selecting 'Request changes', otherwise 'Approve'.

User Experience

• The feature/bugfix is self-documenting from within the product.
• The change provides the end user with a way to fix their problem (no dead ends).

Functionality

• The code works and performs its intended function with the correct logic.
• Performance has been factored in (for example, the code cleans up after itself to not cause memory leaks).
• The code guards against edge cases and invalid input and has tests to cover it.

Maintainability

• The code is readable (too many nested 'if's are a bad sign).
• Names used for variables, methods, etc, clearly describe their function.
• The code is easy to understood and there are relevant comments explaining.
• New algorithms are documented in the code with link(s) to external docs (flowcharts, w3c, chrome, firefox).
• There are comments containing link(s) to the addressed issue (in tests and code).

Quality

• The change does not reimplement code.
• There's not a module from the ecosystem that should be used instead.
• There is no redundant or duplicate code.
• There are no irrelevant comments left in the code.
• Tests are testing the code’s intended functionality in the best way possible.

Internal

• The original issue has been tagged with a release in ZenHub.

[bahmutov]

Added Desktop GUI tab showing experiments

[bahmutov]

UI review

Desktop GUI experiments tab

"Learn more" link goes to (non-existent yet) https://on.cypress.io/experiments

Experiment enabled

No experiments enabled

• looks good, @brian-mann ?

[bahmutov]

UI Review

Headless run with experiments enabled (without experiments same terminal output)

• looks good, @brian-mann ?

[jennifer-shehane]

I'm gonna work on the Desktop-GUI to freshen the design up for the experiements flags display.

[bahmutov]

Feedback - changes requested

• instead of creating new wrapper and only accepting environment variable, put experiments into the existing config. That way the flags can be passed in cypress.json, CLI, environment variable or overwritten by the plugins. When you change the cypress.json, the test runner will reload, just like now, and would not require every user setting an environment variable.
• name experimental properties in the config starting with experimental like experimentalComponentTesting. This way the Experiments settings tab can still show them nicely

[brian-mann]

@bahmutov I would camel case the experiments...

experimentalComponentTesting
vs
experimental_componentTesting

[bahmutov]

Updated logic to just use config, still have to clean up code a little bit from previous code before pushing.

Experimental properties are just keys in the config

The experiments tab in Desktop GUI picks those properties and shows them nicely if they are non-default

Terminal output

[jennifer-shehane]

When we talked about this earlier, I assumed this would be a nested object, which seems cleaner to me than a bunch of very long config values starting with experiemental, but....I guess this complicates things when wanting to just turn it on with one env var.

"experiments": {
  "componentTesting": true,
}

Looked at some other tools cause I wasn't totally satisfied.

• Docker seems to have you set experimental: "enabled", and then just list the features you want out separately, which I don't think is great either. https://docs.docker.com/engine/reference/commandline/cli/#experimental-features
• Node does have everything prefixed with experimental like --experimental-json-modules https://nodejs.org/api/cli.html
• Chrome also has an experimental flag outright to turn them all on or off and then separate flags which are all prefixed experimental. https://developer.chrome.com/extensions/experimental
• Kubernetes uses feature-gates 😆 https://kubernetes.io/docs/reference/command-line-tools-reference/feature-gates/ Although they do have a nice feature where you can say, in the CLI print all of the experimental features available (probably overkill for us know though, they have 100's of features).
• Bazel has experimental prefix https://docs.bazel.build/versions/2.0.0/command-line-reference.html
• Parcel experiemental prefix.
• Everything else I saw doesn't explicitly specify 'experiemental' as part of actually setting the value and solely has it mentioned in documentation.

I think what looks weird is the camelCasing. Nobody camelCases the values but I get wanting to stick with our camelCase convention.

[jennifer-shehane]

Clicking into the Experiments panel completely borks if you do not have any experiments defined. I've updated the title to be [WIP] as this is not ready for review.

[jennifer-shehane]

My first iteration. I decided to pull inspiration from the Dashboard, tried to get a similar design for their Beta features - the main difference being they call theirs 'Cypress Labs' and ours is called 'Experiments'.

Current Design

Dashboard

[jennifer-shehane]

I started to write test in the Desktop Gui specs for the Experiments panel, but not nearly enough. I also updated the code to iterate through 'experiments' and not blow up when there are no experiment variables defined.

This code is a bit naive. We won't be able to pull any key starting with experimental, we will have to explicitly whitelist all keys because people could just make up keys and it would do nothing. https://github.com/cypress-io/cypress/blob/experiments-6257/packages/server/lib/experiments.ts#L35:L35

We also need to build up more properties for each experiment for the display:

• title/name
• value
• default value
• description

[bahmutov]

@jennifer-shehane did you push any code? I don't see any commits. I love the UI with toggle - but these are read-only, right? since the user has to change them using cypress.json or CLI or env for now?

We still control the experiments, thus the user cannot just put any experimentalX config value, we still whitelist keys, and only take the ones we know about.

As far as props for experiments: 'title/name', 'value', 'default value', 'description' - we have value coming from config object (resolved), title and description are the text in experiments.jsx and the only missing thing I think is default value. Unfortunately, since the experiment is part of config, our config has to have resolved objects with current value and defaultValue and from fields. We can tackle this in a separate PR - the config coffeescript code is already hairy enough I feel, I would

• land this PR after more tweaks and tests
• rewrite config in TypeScript (at least the interface)
• add default value property to each resolved config value

[bahmutov]

@jennifer-shehane I have reworked the PR to be more to your liking - all experimental summaries and names are in a single place in server/lib/experiments.ts file and experiments.jsx is all dynamic and nothing is hardcoded.

[bahmutov]

• I have hidden the "Experiments" panel if there are no experiments
• the complex config objects - I think it makes sense to look at this in the components testing PR, since it will be a concrete question there

[jennifer-shehane]

Updated the styles of the ON/OFF switch, so new design up for feedback.

[jennifer-shehane]

Overall this PR looks good to me, barring approval for the design changes on the 'Experiments' panel