Old build is run in watch mode

[korya]

• Operating System: MacOS
• Node Version: v10.14.1
• NPM Version: 6.4.1
• webpack Version: 4.27.1
• karma-webpack Version: 4.0.0-rc.5

Scenario

1. Run karma in watch mode: npm run test
2. Wait for the bundle to be built and tests to run. The tests should succeed.
3. Now modify the spec file such that the tests would fail: sed -i 's/toBe([0-9])/toBe(9)/p' src/one.spec.js.

Expected Behavior

After step 3, the new bundle should be run and fail:

HeadlessChrome 71.0.3578 (Mac OS X 10.13.6) one should equal 1 FAILED
	Error: Expected 1 to be 9.
	    at <Jasmine>
	    at UserContext.it (webpack:///./src/one.spec.js?:7:58)
	    at <Jasmine>
HeadlessChrome 71.0.3578 (Mac OS X 10.13.6): Executed 1 of 1 (1 FAILED) ERROR (0.003 secs / 0.003 secs)

If re-run npm run test command, I see the expected result: the tests fail.

Actual Behavior

Although I can see that webpack recompiled the bundle and the reported hash changed, the build succeeds as if it would run the original (unmodified) tests.

Code

// karma.conf.js
module.exports = config => {
  config.set({
    frameworks: ['jasmine'],

    files: [ 'index.js' ],

    preprocessors: {
      '*.js': [ 'webpack' ],
    },

    webpack: {
      entry: './index.js',
      mode: 'development',
    },

    webpackMiddleware: {
      // stats: 'minimal',
    },

    reporters: ['progress'],
    port: 9876,
    colors: true,
    logLevel: config.LOG_INFO,
    autoWatch: true,
    browsers: ['ChromeHeadless'],
    singleRun: false,
    concurrency: Infinity,
  });
};

// index.js
const testCtx = require.context('./src', true, /\.spec\.js$/);

testCtx.keys().forEach(testCtx);

// src/one.js
export default 1;

// src/one.spec.js
import one from './one';

describe('one', () => {
  it('should equal 1', () => {
    expect(one).toBe(9);
  });
});

How Do We Reproduce?

1. Checkout the repo: https://github.com/korya/karma-webpack-watch-mode-issue
2. Run the scenario steps:
   1. Run karma in watch mode: npm run test
   2. Wait for the bundle to be built and tests to run. The tests should succeed.
   3. Now modify the spec file such that the tests would fail: sed -i 's/toBe([0-9])/toBe(9)/' src/one.spec.js.

Full Log

$ npm run test

> webpack-context-require@1.0.0 test /Users/korya/dev/webpack-context-require
> karma start ./karma.conf.js

13 12 2018 14:08:29.056:WARN [karma]: No captured browser, open http://localhost:9877/
ℹ ｢wdm｣: Hash: da049fc7bf26ee608e7b
Version: webpack 4.27.1
Time: 15ms
Built at: 2018-12-13 2:08:29 p.m.
   Asset      Size  Chunks             Chunk Names
index.js  5.81 KiB   index  [emitted]  index
 main.js  5.81 KiB    main  [emitted]  main
Entrypoint main = main.js
Entrypoint index = index.js
[./index.js] 97 bytes {main} {index} [built]
[./src sync recursive \.spec\.js$] ./src sync \.spec\.js$ 178 bytes {main} {index} [built]
[./src/one.js] 18 bytes {main} {index} [built]
[./src/one.spec.js] 116 bytes {main} {index} [optional] [built]
ℹ ｢wdm｣: Compiled successfully.
13 12 2018 14:08:29.072:INFO [karma-server]: Karma v3.1.3 server started at http://0.0.0.0:9877/
13 12 2018 14:08:29.072:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
13 12 2018 14:08:29.076:INFO [launcher]: Starting browser ChromeHeadless
13 12 2018 14:08:29.448:INFO [HeadlessChrome 71.0.3578 (Mac OS X 10.13.6)]: Connected on socket PCQ1DpJMsd9hi8fGAAAA with id 22158802
HeadlessChrome 71.0.3578 (Mac OS X 10.13.6): Executed 1 of 1 SUCCESS (0.001 secs / 0.001 secs)
TOTAL: 1 SUCCESS


##### Run 'sed -i 's/toBe([0-9])/toBe(9)/' src/one.spec.js



ℹ ｢wdm｣: Compiling...
ℹ ｢wdm｣: Hash: 740e0db875c82d372f14
Version: webpack 4.27.1
Time: 13ms
Built at: 2018-12-13 2:09:13 p.m.
   Asset      Size  Chunks             Chunk Names
index.js  5.88 KiB   index  [emitted]  index
 main.js  5.88 KiB    main  [emitted]  main
Entrypoint main = main.js
Entrypoint index = index.js
[./index.js] 97 bytes {main} {index}
[./src sync recursive \.spec\.js$] ./src sync \.spec\.js$ 178 bytes {main} {index} [built]
[./src/one.js] 18 bytes {main} {index}
[./src/one.spec.js] 141 bytes {main} {index} [optional] [built]
ℹ ｢wdm｣: Compiled successfully.
HeadlessChrome 71.0.3578 (Mac OS X 10.13.6): Executed 1 of 1 SUCCESS (0.001 secs / 0 secs)
TOTAL: 1 SUCCESS

^C
$ npm run test

> webpack-context-require@1.0.0 test /Users/korya/dev/webpack-context-require
> karma start ./karma.conf.js

13 12 2018 14:09:26.554:WARN [karma]: No captured browser, open http://localhost:9877/
ℹ ｢wdm｣: Hash: 740e0db875c82d372f14
Version: webpack 4.27.1
Time: 14ms
Built at: 2018-12-13 2:09:26 p.m.
   Asset      Size  Chunks             Chunk Names
index.js  5.88 KiB   index  [emitted]  index
 main.js  5.88 KiB    main  [emitted]  main
Entrypoint main = main.js
Entrypoint index = index.js
[./index.js] 97 bytes {main} {index} [built]
[./src sync recursive \.spec\.js$] ./src sync \.spec\.js$ 178 bytes {main} {index} [built]
[./src/one.js] 18 bytes {main} {index} [built]
[./src/one.spec.js] 141 bytes {main} {index} [optional] [built]
ℹ ｢wdm｣: Compiled successfully.
13 12 2018 14:09:26.570:INFO [karma-server]: Karma v3.1.3 server started at http://0.0.0.0:9877/
13 12 2018 14:09:26.570:INFO [launcher]: Launching browsers ChromeHeadless with concurrency unlimited
13 12 2018 14:09:26.579:INFO [launcher]: Starting browser ChromeHeadless
13 12 2018 14:09:26.948:INFO [HeadlessChrome 71.0.3578 (Mac OS X 10.13.6)]: Connected on socket -OKokGONWWP4Z2SPAAAA with id 24758883
HeadlessChrome 71.0.3578 (Mac OS X 10.13.6) one should equal 1 FAILED
	Error: Expected 1 to be 9.
	    at <Jasmine>
	    at UserContext.it (webpack:///./src/one.spec.js?:7:58)
	    at <Jasmine>
	Error: Expected 1 to be 9.
	    at <Jasmine>
	    at UserContext.it (webpack:///./src/one.spec.js?:8:58)
	    at <Jasmine>
HeadlessChrome 71.0.3578 (Mac OS X 10.13.6): Executed 1 of 1 (1 FAILED) ERROR (0.002 secs / 0.002 secs)
^C

[korya]

Update

If I modify the entry file, touch index.js, karma runs the newly built bundle (just as expected). That is, the following scenario:

1. Run karma in watch mode: npm run test
2. Wait for the bundle to be built and tests to run. The tests should succeed.
3. Now modify the spec file such that the tests would fail: sed -i 's/toBe([0-9])/toBe(9)/p' src/one.spec.js.
4. Run touch index.js.

results in test failure as expected.

[matthieu-foucault]

Are you using karma v3.1.3? If so this is not a karma-webpack bug. See karma-runner/karma#3226

[korya]

OK. Thanks. Indeed v3.1.1 works just fine. I am closing the issue.