Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat: support snapshots and console props within multi-domain #20949

Merged

Conversation

AtofStryker
Copy link
Contributor

@AtofStryker AtofStryker commented Apr 6, 2022

User facing changelog

N/A

Additional details

The goal of this PR is to get snapshots and consoleProps working from within the cy.origin context after the spec file has been run in open mode. Currently, snapshots do NOT work within multi origin since snapshots are captured within the specbridge for the cross origin aspect of the test.

running the basic_login_spec with snapshots

snapshot-multi-origin-1

running the basic_login_spec with consoleProps

consoleProps-multi-origin-1

Methods Explored for Multi Origin Snapshotting/ConsoleProps

  1. Having the spec bridge render the snapshot and consoleProps on request from the runner

    This is a bit of a complicated problem, mostly relating the the nature of dealing with CORS policy and postMessage() serialization. Because of these known issues, the first technique to get this working was exploring the path of least resistance and attempting to ask the appropriate spec bridge for the snapshot after spec has been run for the pinned snapshot, and have the specbridge show the appropriate snapshot and print out the consoleProps. Sounds easy enough, right? Well, not quite.

    When attempting this method, I ran into the following problems:

    • CORS policy : Probably the biggest factor, but asking the spec bridge to render a snapshot means the spec bridge MUST have access to the document. Take the following example:
      • I start my test fitting an origin policy of my BASE_URL / primary domain. The AUT iframe src attribute fits the origin policy of my BASE_URL primary domain.
      • I then navigate to https://www.foobar.com with a corresponding cy.origin(). The AUT iframe has now changed it's src attribute to fit an origin policy pertaining to https://www.foobar.com.
      • My AUT navigates back to an origin policy matching that of my BASE_URL primary domain. My spec now finishes.
      • I attempt pin a snapshot from https://www.foobar.com as well as the command log's consoleProps for the given snapshot. I ask the spec bridge to render this log's snapshot on the page. This ultimately fails because the AUT is in a different origin, and the document the spec bridge has access to is stale, IE, detached. I would need to set the src on the AUT to the proper origin that needs to render the snapshot, which triggers a load event. I would then need to try and stop said load event with window.stop(), which would need to be invoked in the origin that has ownership of the resource. This causes race conditions for content being loaded on the page, leading to "flickering" and a few other odd experiences, as well as some performance issues. This would also require pretty significant intervention of how we handle page load events, since the page never exactly loads when we stop the load early.
    • Determinism of "final" snapshot state and CORS policy: Besides the issues with CORS, the runner needs to have a 'final state' snapshot to render when the snapshot is unpinned. When the spec file is finished running, the AUT may or may not have ended in a cross origin state. We can't feed this 'final state' snapshot to the runner if the AUT ends in a cross origin state.
    • Performance / UX: as mentioned above, attempting to trigger load events, and then stopping them, leads to some wonky race condition behavior, as well as some weirdness in our page loading behavior. The behavior isn't always deterministic, and sometimes causes some slower performance when attempting to render a snapshot. It would also delay the consoleProps output, as the correct snapshot would need to be rendered on the page in order to show the correct elements. Due to the non deterministic nature, sometimes state DOM elements would be received and wouldn't actually show up in the snapshot.
  2. Serializing the DOM/consoleProps
    This lead me to believe that another course of action needed to be taken. Instead of trying to ask the spec bridge to render snapshots and passing around AUT ownership, I figured it might make sense to explore trying to serialize the DOM for the given snapshot, as well as the consoleProps for the log message. This happens while the test runs, as opposed to on request. When a cross-origin log is then pinned, we can set the src to about:blank effectively by removing it, which occurs incredibly fast and fit's the same-origin policy of the runner. We then take our cross origin snapshot that has been reified, along with it's consoleProps, and display it on the page. This happens quickly as the snapshot has been reified and stored in the primary domain.
    The strange caveat to this method is if our spec ends in a cross origin state. In this case, we ask the appropriate spec bridge to take a snapshot of what is currently on the page, and sent it back to the runner to store as 'final state'. This snapshot is then rendered when a snapshot is not active or unpinned, but is rendered in a same origin context.
    The consoleProps, in order to match the correct element on the page, need to be calculated AFTER the snapshot is thrown on the page. When real time serialization occurs while the test is running, these consoleProp elements are defined as ES6 getters that lazily calculate the values output. This means the output is calculated when the consoleProps are invoked in the console, which is AFTER the snapshot has been rendered.

    For DOM element serialization, two methods were explored:

    • Fully traverse each DOM element to build a 'DOM Tree' that is fully stateful to reify in the primary. Though this method sounds great on paper, it becomes pretty clear that this method isn't performant when working with medium/large size DOMS. Since the tree is build recursively, performance ceilings are quickly hit and the test runner would start to exhibit serious performance problems, sometimes even crashing.
    • Grab the innerHTML of the DOM element. This method is performant and simple, but lacks stateful representation of the element in multiple cases. In an attempt to remedy the state issues, certain characteristics of the element are stored alongside its HTML. Since consoleProp/snapshots that are being serialized back to the primary are all copies, we can inject values into the HTML to make the DOM seem more stateful. EX: setting the value attribute for input elements in the DOM to accurately represent the state of the input at that time when the snapshot is rendered in the primary. This is not a fullproof method and is a bit naive, but the hybrid approach allows us to obtain decent performance benefits as well as most important stateful items. It is also a lot more deterministic compared to other approaches, even though the solution isn't super elegant. This is the implementation that exists in this PR.

Known Issues with this implementation

There are still some stateful issues being worked out that need to be addressed, such as :
*select elements having the correct selected option at snapshot time. There are likely other stateful DOM issues present that need to be discovered
* XHR requests, when presented with fully qualified URLs, take snapshots of the primary domain document, when they should be taking snapshots of the "active" domain
* Some snapshots, in transition to the new origin, take snapshots of the old origin. This is likely a race condition from when the cy.origin function is invoked vs when the new origin is actually loaded. This issue is common

Cy-in-Cy Tests

Since cy-in-cy tests are not available until the 10.x.x release, none are added in this branch. The snapshots directory tries to validate the correct state of the consoleProps against itself. This isn't create because we can't actually see the DOM element output from the consoleProps matched against the snapshot on the page. When available, several cy-in-cy tests should be added to verify correct snapshot behavior. It will also make surfacing and solving known issues a bit clearer.

How has the user experience changed?

User's can now see snapshots within cy.origin invocations.

PR Tasks

  • Have tests been added/updated?
  • Has the original issue (or this PR, if no issue exists) been tagged with a release in ZenHub? (user-facing changes only)
  • Has a PR for user-facing changes been opened in cypress-documentation?
  • Have API changes been updated in the type definitions?
  • Have new configuration options been added to the cypress.schema.json?

@cypress-bot
Copy link
Contributor

cypress-bot bot commented Apr 6, 2022

Thanks for taking the time to open a PR!

@AtofStryker AtofStryker changed the title Md support snapshots and console props feature: support snapshots and console props within multi-domain Apr 6, 2022
@AtofStryker AtofStryker changed the title feature: support snapshots and console props within multi-domain feat: support snapshots and console props within multi-domain Apr 6, 2022
ryanthemanuel and others added 13 commits April 7, 2022 16:06
further progress with getters

cleaned up log/snapshot serialization

attempt to pass and hydrate value state of serialized dom elements

temp commit traversal dom

by some stretch of a miracle this is working...

still somehow working

after massive performance issues with full tree serialization, fix here is to just attach values to inputs for reifying on primary

now we are cookin

test WIP

tests WIP

working multi-domain actions snapshots tests

added more tests to verify snapshots

add tests and refactor certain tests to make simpler

added misc snapshot tests

add navigation snapshot placeholder

add network request snapshot tests

add shadow querying snapshot tests

update test names

added snapshot querying spec

added screenshot snapshot test

add spies,clocks, and stubs tests

implement snapshot tests for traversal commands

rename local storeage snapshot tests to fit convention

add viewport snapshot tests

rename snapshot traversal to fit naming convention

add snapshot waiting tests

added window snapshot tests

implement navigation snapshot tests now that sinon proxy issues internal to log are now fixed

refactor multi-domain snapshot tests to leverage utility method over redefining in each spec
…s to what is going on in this 'here be dragons' code
@AtofStryker AtofStryker force-pushed the md-support-snapshots-and-console-props branch from a597d05 to 42186f1 Compare April 7, 2022 20:09
@AtofStryker AtofStryker force-pushed the md-support-snapshots-and-console-props branch from b06f58b to be4bc19 Compare April 7, 2022 20:38
@cypress
Copy link

cypress bot commented Apr 7, 2022



Test summary

8889 0 102 0Flakiness 0


Run details

Project cypress
Status Passed
Commit 8d3ae17
Started Apr 22, 2022 2:19 PM
Ended Apr 22, 2022 2:32 PM
Duration 12:23 💡
OS Linux Debian - 10.10
Browser Multiple

View run in Cypress Dashboard ➡️


This comment has been generated by cypress-bot as a result of this project's GitHub integration settings. You can manage this integration in this project's settings in the Cypress Dashboard

@AtofStryker
Copy link
Contributor Author

I think I have all comments addressed (hopefully I didn't miss anything). I am going to lok into the firefox 93 serialization errors and see if I can have an update soon!

…s the document is in a cross origin context in the primary, which means accessing any elements will not work
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants