CLI / Extensions Server End-to-End Tests

[dmhenry]

This issue is the result of the Argo Team's July Hack-a-thon. The goal was to have end-to-end testing for at least one shopify-cli-extensions flow, starting with the CLI 3.0. Due to the lack of end-to-end tests in the CLI, however, the scope of these two days ended with compiling manual steps needed to serve an app using a Spin environment. Origin: Testify: cli-extensions.

Need Spin instance so as to avoid logging into a Dev Store:

1. local> spin up extensions-sandbox-app --name $SPIN_INSTANCE
2. spin> cd shopify && SHOP_ID=1 BETA=checkout_one_argo_extensions bin/rake dev:betas:enable
3. local> spin open Go to Partners Internal and create a working Dev Store
4. local> SPIN_INSTANCE=$SPIN_INSTANCE dev spin yarn create-app --local --name $APP_NAME --template {node|php|ruby}
5. local> SPIN_INSTANCE=$SPIN_INSTANCE dev spin yarn shopify app scaffold extension --name=$EXTENSION_NAME --type={product_subscription|checkout_ui_extension|checkout_post_purchase|web_pixel_extension|pos_ui_extension|product_discounts|order_discounts|shipping_discounts|payment_methods|shipping_rate_presenter} --template={vanilla-js|react} --path $APP_DIR
6. local> cloudflared tunnel --url http://localhost:3000 (See https://github.com/Shopify/shopify/issues/365272)
7. local> SPIN_INSTANCE=$SPIN_INSTANCE dev spin yarn shopify app dev --tunnel-url $CLOUDFLARED_TUNNEL --path $APP_DIR --store $DEV_STORE --api-key $APP_API_KEY

Relevant resources:

• CLI Next on Spin
• Test Auth Flows

[byrichardpowell]

Things to test

The create command. We can test this by:

1. Run the create command with all required args (template & name)
2. Check shopify.ui.extension.toml exists
3. Check name & type inside shopify.ui.extension.toml

The dev command. We can test this by:

1. Run the serve command without specifying the tunnel URL
2. Parse CLI output
3. Expect the Extension heading & Preview URL (see screenshot)
4. Expect a Request to that URL to be between 200 & 400

The build command. We can test this by:

1. Delete the extensions/my-extension-name/dist folder
2. Run the build command
3. Check extensions/my-extension-name/dist/main exists

[vividviolet]

The create command. We can test this by:

1. Run the create command with all required args (template & name)
2. Check shopify.ui.extension.toml exists
3. Check name & type inside shopify.ui.extension.toml

4. Check src/index.[tsx|ts|jsx|js] to verify that it exists and that it matches the chosen template (ex. tsx for typescript-react)

The dev command. We can test this by:

1. Run the serve command without specifying the tunnel URL
2. Parse CLI output
3. Expect the Extension heading & Preview URL (see screenshot)
4. Expect a Request to that URL to be between 200 & 400

Why do we expect the response to be between 200 - 400? Can we be more specific? I think it's a 302 redirect response

Can we also verify that calling fetch on the Dev Server url returns a JSON manifest with version, socket.url, console.url and extensions?

Can we also verify that making a request to the Dev Console url returns a 200?

[byrichardpowell]

Check src/index.[tsx|ts|jsx|js] to verify that it exists and that it matches the chosen template (ex. tsx for typescript-react)

We didn't realize when we wrote the last comment, but this is already done.

Why do we expect the response to be between 200 - 400? Can we be more specific? I think it's a 302 redirect response

Yeah, testing the exact response would be better. We just didn't know what it would be when we wrote the issue.

Love the additions to test the dev server. Will try!

[dmhenry]

We did not implement an end-to-end test for the dev command because we would need to couple the test to a working Dev Store with the app installed. Does anyone have thoughts on how we can achieve this test, or are we in agreement that the barrier to setting up this test (and the resulting flakiness) maybe renders it not worth the effort?

CC: @vividviolet @pepicrft @isaacroldan @alvaro-shopify @byrichardpowell

[isaacroldan]

we would need to couple the test to a working Dev Store with the app installed

Do we? The app doesn't need to be installed to execute dev, the only difference is that the link will redirect you to the install page instead of the app page.

What do we want to test in this case? Can/Should we mock all API interactions? to test just that the communication between the cli and the extensions-server works as expected? 🤔

[byrichardpowell]

What do we want to test in this case? Can/Should we mock all API interactions? to test just that the communication between the cli and the extensions-server works as expected?

If we are ok mocking the state of the CLI so that it doesn't prompt the user for input then we would want to test everything in my comment & Trish's comment above. We just weren't sure if mocking was an ok path in this case.

[isaacroldan]

I discussed mocking with @pepicrft a few weeks ago because we think it will be needed for more tests in the future. Having a spin instance with some test data to connect to is possible I guess, but that would be way more complex.

In the acceptance tests we could have a feature to create a "fake environment" with auth tokens, org, app, etc... if you pass an api-key and a store to dev no prompt will be triggered, so we would only need to mock all API calls after that (not that many, mainly the one used to retrieve the whole app data)

Let me bring the mock vs no-mock debate to the cli team so that we are all aligned

[byrichardpowell]

Sounds good, thanks Isaac.