Grand Testing Unification

[rwjblue]

Rendered

[rtablada]

@rwjblue

I do think that getOwner(this) would be best to remain consistent though.

[mmun]

getOwner(this) does not seem consistent to me at all. The test module is not created by the owner and thus is not owned by it. In fact, the reverse is true. The test module creates the owner. The test module has a reference to the owner and chooses to expose it on this.owner.

[mixonic]

The test module is not created by the owner and thus is not owned by it.

Well put @mmun. I agree.

[courajs]

moduleForComponent?

[mmun]

moduleForIntegration is correct. There is nothing specific to components about it. It can be used to test template helpers, or interactions between components. Think of it like a micro acceptance test.

On the other hand, unit testing a component can be done with moduleForUnit, though I recommend sticking with integration tests unless you have a specific reason to unit test.

In short, we no longer need a moduleForComponent and keeping it (other than for backwards compatibility) would cause confusion.

[courajs]

I thought this is an existing test using the old way. In any case, it doesn't match with moduleForComponent below on line 390, which I think it was meant to

[mmun]

You're totally right. My mistake!

[rwjblue]

Ya, overeager find/replace. Fixing....

[kategengler]

👏

My only thought/concern is around https://github.com/rwjblue/rfcs/blob/42/text/0000-grand-testing-unification.md#registering-custom-waiters

I've seen apps where waiters are registered in code (though it never feels right, but often does seems to be the easiest way to get at the conditions that matter for waiting), and the tests are unaware of the custom waiters. liquid-fire does that here https://github.com/ember-animation/liquid-fire/blob/master/addon/transition-map.js#L106 In the case of addons registering waiters, it does seem nice that each test/app doesn't need to know to register the waiter.

[tim-evans]

For some references to code in the wild, I'm using the following code for testing:
https://gist.github.com/tim-evans/5982357f7df5c0c045dd, which follows this RFC in some places.

[davewasmer]

@kategengler this RFC for adding something like run.callback() might be relevant as well (the addon spike uses registerWaiter internally)

[rwjblue]

@katie:

I've seen apps where waiters are registered in code (though it never feels right, but often does seems to be the easiest way to get at the conditions that matter for waiting), and the tests are unaware of the custom waiters.

Yes, I have seen this as well, but I generally feel that modifying your app code to registerWaiters itself is not good.

I could easily see adding a mechanism for our test setup code to invoke a method on each addon so that the addon could do some custom setup. This would allow them to register custom helpers, waiters, and QUnit assertions. However, I also really like knowing where these things are coming from by registering them in the shared test/helpers/module-for-*.js file.

---

I will add an "unanswered question" for this, so we do not lose track of this concern.

[Gaurav0]

Wow. I just have to say, this is going to mean a lot of upgrading.

First strongly consider (and add to alternatives) adding async helpers to unit/integration tests but not removing anything.

I would strongly suggest:

1. Do not do this until Ember is addonized. Once blueprints are in the ember package things will be easier.
2. Make a plan to introduce all of the new features incrementally. Do not make this a "big bang" in a single minor version of Ember / ember-cli.
3. Ensure that the parts requiring changes to ember-cli, ember-qunit, and ember-mocha are introduced in a way that allows all versions of Ember 1.12 - 3.0+ are forward and backward compatible.
4. I still think the regsiterWaiter API is too low level. Look at the ember.run.callback PR for ideas on how to improve
5. Don't wait until 3.0 is almost here to start deprecating stuff. Make a plan to deprecate features 2 to 3 minor releases after their replacements have landed.
6. Document, document, document. And then document some more.

[Gaurav0]

@stefanpenner While that may be true as far as enabling these features goes, I don't see how this is possible (especially without heavy use of reopen) without modifying ember, ember-qunit, and ember-mocha. ember-cli may not require heavy modification if ember is addonized first.

[stefanpenner]

First strongly consider (and add to alternatives) adding async helpers to unit/integration tests but not removing anything.

the current model is a pretty big hazard, but should be supported until 3.0. Transitioning new apps and users away seems prudent. Post 3.0 we can swap, extracting the current to an add-on etc.

You raise a good point though (i think), and that is we should ensure it is possible for both to run in the same code-base at the same time, to ease upgrades.

[stefanpenner]

ember-cli may not require heavy modification if ember is addonized first.

This work is basically done (pending using @trabus's new test infrustrature + rebase), we can safely assume it will land before.

[kategengler]

@rwjblue Somewhere for addons to hook in would alleviate part of my concern. I agree that modifying app code to incorporate registerWaiter is not good but I haven't seen good alternatives for when you need to get at state inside of the app to know whether to wait. I need to look for concrete examples of when I've encountered it (I may no longer have access to those apps), but if I remember correctly it comes up a lot with custom data solutions and with animations.

@davewasmer Thanks for the pointer to the RFC, I do think there are uses for registerWaiter currently that aren't covered by that or the addon, though.

[backspace]

One example is PouchDB-backed Ember applications, which can’t rely on monitoring Ajax calls to know that operations are complete. ember-cli-test-model-waiter has helped me in that sense, but I still have weird intermittent test failures that are probably related to not having proper ordering. I have in the past used a custom helper that waits on a promise set in a controller but it makes me feel dirty 😁

[rwjblue]

@Gaurav0:

First strongly consider (and add to alternatives) adding async helpers to unit/integration tests but not removing anything.

Nothing is proposed to be removed in this RFC. As stated in the RFC, the plan is to implement everything as new additive API that can be consumed and does not propose changing any existing API's.

Do not do this until Ember is addonized. Once blueprints are in the ember package things will be easier.

The changes here are not related to Ember version, and will not live in the Ember repo.

Make a plan to introduce all of the new features incrementally. Do not make this a "big bang" in a single minor version of Ember / ember-cli.

As I state above and in the RFC, these changes will be additive. When this RFC is approved/merged we will begin rolling out the new API's proposed here.

Ensure that the parts requiring changes to ember-cli, ember-qunit, and ember-mocha are introduced in a way that allows all versions of Ember 1.12 - 3.0+ are forward and backward compatible.

The reason this will be implemented in a separate library instead of Ember itself is so that we can implement exactly this sort of cross version support. As mentioned in the unanswered questions section, the lowest Ember version that would be supported has not been determined, but I would prefer 1.12.

I still think the regsiterWaiter API is too low level. Look at the ember.run.callback PR for ideas on how to improve.

I will await a decision from #115 before embedding those changes here, however it should be very straightforward to implement whatever solution is decided there.

Don't wait until 3.0 is almost here to start deprecating stuff. Make a plan to deprecate features 2 to 3 minor releases after their replacements have landed.

As stated in the "Migration Plan" section of this RFC, the deprecation of the existing testing infrastructure will be completely based on feedback of the new system. If things are going well, we could start deprecating existing features within a minor version or two of when these changes land. I will update that section to put a clearer timeline there.

Document, document, document. And then document some more.

Indeed.

[rwjblue]

@stefanpenner:

You raise a good point though (i think), and that is we should ensure it is possible for both to run in the same code-base at the same time, to ease upgrades.

All existing API's will remain unchanged and will be able to run alongside and independent of the new proposed API's. It will absolutely take time to upgrade (even if my hopes of some automation assistance come to fruition), and we need to ensure that existing test suites continue to work.

[Gaurav0]

@rwjblue

Nothing is proposed to be removed in this RFC. As stated in the RFC, the plan is to implement everything as new additive API that can be consumed and does not propose changing any existing API's.

Not true.

From the RFC

Based on feedback received from the community on the usability of the new structure proposed here, we will deprecate usage of existing Ember API's:

Ember.Test.registerHelper
Ember.Test.registerAsyncHelper
Ember.Application#setupForTesting
Ember.Test.unregisterHelper
Ember.Test.onInjectHelpers
Ember.Application#injectTestHelpers
Ember.Application#removeTestHelpers
Ember.Test.registerWaiters
Ember.Test.unregisterWaiters
Ember.Test.*
existing ember-qunit / ember-mocha API's.

I suggested, as an alternative, only adding unit/integration async test helpers, not unifying them, and not deprecating any of the above mentioned APIs.

The changes here are not related to Ember version, and will not live in the Ember repo.

My understanding from your and @stefanpenner 's previous statements is that all of the blueprints are planned to be moved from the ember-cli repo to the ember repo as part of addonizing the ember repo, so that they can be correct according to the ember version being used.

[rwjblue]

@Gaurav0:

Nothing is proposed to be removed in this RFC. As stated in the RFC, the plan is to implement everything as new additive API that can be consumed and does not propose changing any existing API's.

Not true.

Nowhere in the quoted section do I speak of "removing" anything. I speak of deprecating once we are generally happy with the new solution.

I suggested, as an alternative, only adding unit/integration async test helpers, not unifying them, and not deprecating any of the above mentioned APIs.

I will add that to the alternatives section. However, continuing down the path of wall papering over large problems in our infrastructure seems very bad to me.

[rwjblue]

@kategengler:

liquid-fire does that here https://github.com/ember-animation/liquid-fire/blob/master/addon/transition-map.js#L106

This waiter could be written as such (outside of app code):

// addon-test-support/waiters/running-transitions.js

import { testWaiter } from 'ember-test-helpers';

export default testWaiter(function() {
  let transitionMap = this.owner.lookup('service:transition-map');

  return transitionMap.runningTransitions() === 0;
});

This is roughly the same implementation that exists in liquid-fire today, but as of this moment it would still require manual registration (in the shared tests/helpers/module-for-*.js files). I added a section to "unanswered questions" about either a hook to allow addons to do this work themselves, or automatically registering helpers and waiters across the board (similar to what we do with initializers). I will continue to think/work on that aspect of this RFC...

[kategengler]

@rwjblue I think its a scenario with two gross options -- do you let your app know about your testing framework or do you let your acceptance tests know about the internals of your app? If that way (looking up + reaching in) is generally accepted as kosher and with a hook for addons, I think my concerns are allayed.

[rwjblue]

@kategengler:

I think its a scenario with two gross options -- do you let your app know about your testing framework or do you let your acceptance tests know about the internals of your app?

Yeah, I agree. In this case (liquid-fire), I believe that the test waiter I wrote is just another consumer of a normal public API (the transitionMap.runningTransactions() method). It also has the benefit of not increasing production app size for the benefit of tests.

Regardless, I think we are on the same page. I will try to come up with a nice API for the hook mechanism...

[machty]

Been discussing with @rwjblue but wanted to share a potential tweak to test helper semantics that I think will solve a lot of tricky issues pertaining to route loading substates and other intermediate/transient states that occur while test waiters are still unsettled.

Let's say you have the following:

await this.click('.some-btn');
await this.click('.another-btn');
let $sel = await find('.banner');
assert.equal($sel.text(), "wat");

I don't think the semantics of the above are 100% spelled in the RFC, but most would assume the behavior is: 1) try clicking .some-btn immediately, pause the async fn, and don't resume until all the test waiters settle, and then 2) try clicking .another-btn immediately, and again wait for its effects to settle as reported by all registered test waiters, and then 3) try finding .banner immediately, etc.

In all cases, if the selector (e.g. .some-btn) isn't found right away (synchronously), this throws a failed assertion.

These are the somewhat classic semantics that most people have come to expect from ember-testing, but with these semantics, it's still really hard to test:

1. Route loading substates or any transient states that occur while test waiters are still unsettled
2. Code with long / looping timers

There is a subtle tweak that has been proposed that (I believe) maintains backwards compatibility and should for the most part remain in line with people's mental models. Given the same code above:

1. While test waiters are unsettled, try repeatedly to find the selector (e.g. .some-btn).
2. Once you find the selector, "click" it.
3. Resume the async testing function immediately, without waiting for the effects of the click to "settle" (i.e. resume the function before test waiters have necessarily settled)
4. If test waiters settle before the selector can be found, throw a "Selector Not Found" failed assertion.

I'd call this the "retry-until-settled" semantics. If this seems weird/unusual, keep in mind that if you're structuring tests using the async fn approach proposed in the RFC, it doesn't actually change anything about the behavior of the test, aside from the following benefits:

1. Loading substates are now testable

For instance if you use loading substates in your app, and your loading template has <h1 class="loading-banner">Loading...</h1>, you can now test that loading substate as follows:

await this.click('a.some-link-to-slow-route');
let $loadingBanner = await find('.loading-banner');
assert.equal($loadingBanner.text(), "Loading...");

let $slowRouteBanner = await find('.slow-route-banner');
// this will resume when the slow route finishes loading

Previously, there was no easy way to test loading substates because ember-testing wouldn't run your andThen() or wait() callbacks while test waiters were still running, but with the "retry-until-settled" approach, your tests resume more eagerly, once the selector appears in the DOM.

2. Tests are more immune to timers

Since ember-testing's internal waiters block on unfired Ember.run.later timers, this can lead to frozen tests that eventually time out due to a) long timers or b) polling loops, oftentimes unrelated to the feature under test.

With "retry-until-settled", you can test features even if some background outstanding timer hasn't yet fired.

NOTE: this is only a partial solution to the trickiness surrounding testing timers. This approach does make things easier / faster when tests pass, but it also means that if a selector fails to match, the test will only fail when test waiters settle (or the test case times out). In other words, there's still room for testing abstractions brought to you by ember-lifeline, or by using smaller timer values or breaking timer loops when testing. But keep in mind that this downside is also present in present ember-testing semantics.

timeout option

I think we should also consider an option for find() (and all the helpers that use it) to wait beyond test waiter settlement for an element with matching criteria to appear. This is useful when writing a custom waiter is difficult, annoying, or doesn't really make sense (e.g. what does it mean for a WebSocket connection to "settle").

One possible API might look like:

await this.click('a.some-link', { timeout: 5000 });

The timeout option implies: "if this selector hasn't matched by the time test waiters have settled, wait an additional 5 seconds for it to show up before failing". This essentially opts into "Capybara" semantics, which shouldn't be the default since it slows the TDD cycle (and slows down the test suite when tests are failing).

Admittedly, this option is not as crucial as "retry-until-settled" semantics, but I'd at least want it to be possible to implement such a helper myself without being boxed in by hard-wired waiter logic.

[courajs]

Can "retry-until-settled" be made reliable for testing loading routes? It leaves me feeling a little nervous that the runtime might not "catch" the loader before the model resolves.
Maybe it will be reliable because it'll check on every re-render, but it might inspire more confidence to have an explicit test helper for loading routes.

[machty]

@courajs I don't know what the convention is, or whether there is even one, but I would think most people testing something timing-dependent things like loading routes are already using testing tools to control the resolution timing of model hook promises (perhaps ember-cli-mirage does this?). Something like the following:

let resolveModel = stubModelHooks();
await this.click('a.some-link-to-slow-route');
let $loadingBanner = await find('.loading-banner');
assert.equal($loadingBanner.text(), "Loading...");

resolveModel({ name: "fake model" });

let $slowRouteBanner = await find('.slow-route-banner');

This pattern seems pretty robust against timing dependencies (or at least reduces them to the same robustness as code that resumes after test waiters settle).

That said I do think we might want to consider using something like Mutation Observers to "kick" a paused retrying finder so that by default Ember is catching as many intermediate states as possible, but generally speaking I think a stubbing/mocking controller model resolution solution makes the most sense here.

[jgwhite]

@machty iirc Capybara uses something much like retry-until-settled and it seems to work well. Perhaps worth discussing: they ended up adding helpers of the form assert page.has_no_content?("wat") as subtly distinct from assert !page.has_content?("wat") to deal with the case where you're expecting "wat" to disappear and the latter form will settle early and give a false negative.

[machty]

@jgwhite FWIW I've been spiking out some of the APIs for this RFC in ember-concurrency land here

This is the API I had in mind for what you're describing:

https://github.com/machty/ember-concurrency/blob/4099e09/tests/unit/test-utilities-test.js#L112-L119

Basically, find() would have a count option that defaults to 1 and asserts if the selector finds more or less than the count provided. If you expect an element to disappear, use count: 0. I think this API feels mostly nice, and encourages a fail-early succinctness (though I wonder if a better default when count is unspecified would be to match any number of elements greater than 1, since I can imagine it being hard/impossible/brittle in some testing environments to know up front how many of a certain element might appear, e.g. if you're testing some network that doesn't always return the same number of items).

RE Capybara: they only have that awkward unfortunate distinction for their basic vanilla set of assertions, but the problem goes away when using RSpec. In our case, since all find()ers are async and need to be awaited, we avoid the problem of two syntaxes where one fails synchronously.

Also, to be clear, these insights were inspired by my experiences with Capybara, but the big difference between what I'm proposing and Capybara is that unlike Capybara, we have an established notion of settledness that we leverage to make our assertions/finders fail ASAP. Specifically, the default behavior of the find() helper I'm proposing is that once test waiters have settled (timers have elapsed, routes have full transitioned, etc) an assertion will be thrown if find() still hasn't found a match. Capybara can't do this (or I'm unaware of it) because there's no built-in concept of test waiters. This means that Capybara assertions fail only when some timeout is reached, which makes for a poor/slow experience when doing TDD since you have to wait for that timeout on every failed test assertion. (Also, I found Capybara to be inconsistent with which test helpers / assertions had the waiting behavior; finding elements would wait, but attempting to click them wouldn't, etc; I think we can do better)

[jgwhite]

@machty find with a count sounds awesome to me and might help catch other classes of subtle non-async testing bugs. Totally agree with your thoughts on capybara.

[courajs]

Would find with count: 0 resume the test function immediately as soon as 0 instances of the selector are present? How would I test that, once the transition to a new route has finished, a selector isn't present on the new page?
Using find to force settling seems a little awkward, and the count: 0 means that an assertion core to the test is implicit in find:

await this.click('.delete-all-users');
await this.click('a.nav-to-user-list');
await this.find('.exists-on-user-list');
await this.find('.user', { count: 0 });

[machty]

@courajs I think you've identified the fundamental tradeoff with this API, in that there are definitely still some awkward cases where you do want the settling behavior. I think in this case you have two options:

1. await this.wait() (or perhaps it'll be renamed to await this.settle() to be less awkward/redundant) before finding/asserting
2. Make a positive assertion about the post transition state before making negative (count: 0) assertions.

Option 2 isn't unique to what I'm proposing; there's been times in the past with classic test waiter behavior where I click()ed a button that essentially was no-op-ing, and getting false passing assertions (e.g. "after submitting this form, I expect NOT to see an error banner"). In these cases, a sprinkling of a positive assertion got me back on the right path.

Maybe there's a way we can improve this testing story even further; I just don't see any alternative test-waiter-y API that doesn't suffer from the major issues of making intermediate states untestable (same goes w WebSockets).

an assertion core to the test is implicit in find

I might be misunderstanding you but I actually like that within find() is an implicit assertion. It often entirely removes the need for a separate line of test code to ensure that $sel.length === 1 and feels SQL-y (in a good way) where you give the framework more info up front to be smart about things. And you can always do additional asserts against the selector if need be.

Either way, I think it might be a good idea to publish an addon that let's people try out some of these ideas and see how they actually feel in production apps.

[tchak]

I have one grief already present in current test system and which will amplify with this new proposal. The fact that we will rely on this makes it impossible to write test(() => {}). I know this is basically an aesthetic issue, but could we have some way to access test context through a parameter for these who prefer a more functional style?

test('should be ok', ({ assert, context, helpers }) => {
  context.owner.lookup();
  context.get();
  helpers.find();
  assert.ok();
});

[rwjblue]

@tchak - That is basically a qunit/mocha issue. I know on the QUnit side @martndemus opened an issue a while back about this (I believe the suggestion was to pass the context as a second argument or something?).

[tchak]

@rwjblue I kinda like my idea with hash destructing. But as I said, this is mostly an aesthetic issue. It will be very easy to come up with an addon to make it work the way I want it. So not a big deal.

[bendemboski]

I have a proposal that isn't exactly related to all of this, but might end up being, so I want to mention it here.

One small thorn in my side that has come up a few times is the fact that AFAIK ember-test-helpers doesn't provide support for tests whose subject doesn't have a registered factory. So if my test subject is an object without a registered factory, or is something other than an object, like a mixin (and my test is going to subclass directly), I have 2 options that I'm aware of:

1. Write a test module that doesn't use any of the moduleFor variants, so just a QUnit module without all the isolated container goodness of ember-test-helpers, and then write a bunch of custom code duplicating moduleFor functionality.
2. Pick some random factory as the subject of my test.

I usually opt for number 2:

moduleFor('router:main', 'my mixin or whatever', function (assert) {
});

It doesn't look like it would be that difficult to factor the subject logic out of AbstractTestModule and then create a fourth layer in the module-class hierarchy (we might want a better name):

class AbstractTestModule {
  // as-is today
}

class IsolatedContainerTestModule extends AbstractTestModule {
  // All the code from TestModule except subject-related code
}

class SubjectTestModule extends IsolatedContainerTestModule {
  // identical to TestModule's current functionality
}

class IntegrationOrAcceptanceOrWhateverModule extends SubjectTestModule {
  // as-is today
}

which would allow, for example, a nice way of testing mixins that call into a service via getOwner():

import Ember from 'ember';
import { containerModule } from 'ember-qunit';
import MyCoolMixin from 'my-addon/mixins/cool';

containerModule('my cool mixin', function() {
  needs: [
    'service:something-or-other'
  ]
});

test('it works without a someFunction() override', function(assert) {
  this.register('object:test', Ember.Object.extend(MyCoolMixin));
  let obj = this.container.lookup('object:test');
  Ember.run(() => obj.doSomething());
  let service = this.container.lookup('service:something-or-other')
  assert.equal(service.get('callCount'), 1, 'it called the service');
});

test('it works with a someFunction() override', function(assert) {
  this.register('object:test', Ember.Object.extend(MyCoolMixin, {
    someFunction() {
      return;
    }
  }));
  let obj = this.container.lookup('object:test');
  Ember.run(() => obj.doSomething());
  let service = this.container.lookup('service:something-or-other')
  assert.equal(service.get('callCount'), 1, 'it called the service');
});

That may seem like a long way to go just to avoid including 'router:main', , but I think it makes a lot of sense architecturally.

I've though about working on a PR for this functionality, but that seems silly with the test unification on the horizon, so I wanted to bring this up here to get people's thoughts on it, and see if we want to factor it into the work described here, or if I should just go file an issue in ember-test-helpers, or if anybody has another idea for how to support use cases such as the one I've described.

[rwjblue]

One small thorn in my side that has come up a few times is the fact that AFAIK ember-test-helpers doesn't provide support for tests whose subject doesn't have a registered factory.

Hmm, I believe it does not attempt to lookup a registered object if you specify integration: true or if you specify a factory method in the options argument (the same one that can contain beforeEach or afterEach). Please open an issue over there for that if this doesn't work.

[bendemboski]

@rwjblue I'm not following. If I use moduleFor, I get a TestModule, which sets this.subjectName to be the first argument, and treats this.subjectName as the name of a factory here and here (although I guess the second one won't get triggered without a call to this.subject()). It doesn't look like it's at all expecting to not get a factory name.

Am I really confused here? I'm happy to go file an issue in ember-test-helpers, but it sounds like you're talking about it like it's a bug, and it looks like it was never intended to work that way...

[pixelhandler]

@rwjblue regarding…

Should this.find return a jQuery wrapped element? I would prefer to stick with the main DOM API's here, so that we have a chance to share tests between the Fastboot and Browser.

We worked on https://github.com/cibernox/ember-native-dom-helpers/releases/tag/v0.2.0 cc\ @cibernox and decided to only return HTMLElement or NodeList from find helper. If a test must use jQuery the developer can use jQuery(find('a')) explicitly.

[aureliosaraiva]

@rwjblue I believe that us should to use DOM API NATIVE. I like JQuery, but the Ember needn't it internal. I agree with @pixelhandler, if the developer want to use JQuery him can to use.

[ming-codes]

Being the Grand Testing Unification, don't we need a story for testing node.js code in Ember CLI addons? Or would that be outside the scope of this RFC?

[dwickern]

I'm not sure if anyone has done this already, but I couldn't find it so I made an ember-watson transform: abuiles/ember-watson/pull/112

[mmun]

Awesome @dwickern :)

[mehulkar]

Is this still supposed to be open?

[Gaurav0]

I think this was superceded by RFC #232 .

[rwjblue]

Not fully actually. I'll close for now, but there are still ideas that need to be moved forward.