Integration tests harness for plugins

[glebm]

We need to figure out how to test plugins easily.

For plugins that only affect a small part of Thredded, such as markdown_coderay, unit tests are sufficient, and we only need to make sure that we test with the latest release and the current master.

However, for plugins that change Thredded more extensively it would be nice to run some of the existing Thredded tests and reuse the current Thredded test harness.

One way this can be achieved:

Have the plugin test harness perform the following setup:

1. Add a git submodule of thredded in the plugin (e.g. at spec/support/thredded).
2. Check out the branch of the submodule we want (e.g. master or latest release).
3. Add a path dependency on the plugin code to the spec/support/thredded/thredded.gemspec and spec/support/thredded/spec/dummy/config/application.rb. Hopefully this doesn't create a dependency cycle (thredded depends on the plugin which depends on thredded).
4. Run bundle.
5. Run rails g thredded:my_plugin:install.
6. Modify the intializer.
7. Modify / disable the irrelevant tests (e.g. a plugin that replaces Markdown with WYSIWIG wouldn't want to run the posts contents formatting tests because they would all fail anyway).
8. Symlink the plugin tests from spec/thredded/my_plugin into spec/support/thredded/spec/my_plugin.

With this setup, the integration tests can then be run from within the modified thredded submodule directory. The unit tests can still run as usual.

---

Raised by @timdiggins in #441 (comment):

Do you have any opinions on how thredded could provide testing support for plugins -- would be nice to reuse factories and even the dummy app. But I can't quite work out how it's possible.

(It would also be great if one could run all the thredded tests on your embedded thredded in your main_app, but I think that might be over-ambitious)

[glebm]

@timdiggins That said, I think Notifier plugin tests could just be unit tests for now. The API is tested from Thredded already.

[timdiggins]

@glebm Some thoughts:

1. agree generally, but maybe not submodule (this is a pain, and freezes the version into the plugin) --- instead some kind of configuration of which branches to test against (in travis.yml) and this is then cloned automatically as part of test run on travis / but just a post-checkout step in the plugin either git clone github...thredded/thredded or cd thredded && git fetch and git checkout ${branch}.

2. yes agreed unit tests for e.g. notifier plugin tests.

But still need to set up a database with users, posts and privates -- which means a dummy app, and then factories or similar. Yes could mock out the post and private post, but then insulating the plugin from changes in thredded which is a bad idea. Would rather somehow just use the dummyapp and factories from thredded (choice of branches (e.g as you say, latest release and master)).

So maybe the above approach (with git submodule or simpler git checkout) would be appropriate for notifier plugin, but not using feature tests, but using just the dummy app / harness -- and just run the plugin's own unit tests

[timdiggins]

@glebm what do you think about including the /spec directory within the published gem so that other gems can use the spec files (particularly factories and dummy) -- without having to use submodule or separate checkout and bundle local.thredded

[timdiggins]

NB - this would be only first step to integration test harness, but we can slowly make this happen. For the notifiers plugin I was initially thinking of using factories and DB, but will first of all try a mocking approach and see if that is good enough (though not sure how easy it is to mock an ActiveRecord with a validating double and no db... -- could end up baking in expectations about the fields that could change...)

[timdiggins]

Thinking about this a bit further (after implementing) -- what we really want are objects which are like partial doubles for the key models which we can use as either mocks or stubs in rspec without needing the base models or the db.

These should be tested in the thredded/thredded that the models conform to them (ie. every method they validate exists in the original model). They can then be used in a plugin (and optionally in unit tests too).

Obviously some plugins are going to need the engine and views and db anyway. (that's separate).

Just sketching some ideas for a mock:

require 'thredded/mocks'
include Thredded::Mocks # gives you thredded_mock
let(:post) {thredded_mock(:post) }

describe "'public api'" do
    [:content, :topic, :messageboard].each do |public_field|
      it "should respond to '#{public_field}'" do
        expect(post).to respond_to(public_field) 
      end
    end
  end

  it "works as a validating_partial_double" # etc
end

thoughts @glebm -- or know of a library that will help?

[timdiggins]

better name for thredded_mock ==> thredded_double